Sept 2, 2013
I work in the security industry, so you can probably guess that I am jaded and paranoid. You might guess that I'm depressed, but I'm not. This three-day weekend, I spent time on projects that I enjoy, not leaving the house for any reason except to buy bananas and other necessary food items. Many people in the security industry believe that secure software is impossible. They believe that sometime down the road our best efforts will be overcome by some hacker who wants something that we have. To avoid destruction, we have a process of looking for bugs, making sure that our employees' access rights are limited in some way. But that only goes so far. Sometimes there are serious gaps. So what do you do? Duct tape it together. Vendors sell duct tape by the roll. Then devs, admins and security consultants write Perl or Python scripts to hold everything together and we hope that it doesn't introduce bigger bugs than the ones we're solving.
But I have a different idea in mind. If we put our minds to the task, we can create a secure system. The cost of security is a fraction of the development cost (not dwarfing the development cost like some projects). For a while I thought that Django seemed pretty good at security. It didn't take long for a security bug to show up in a strange place that I didn't have time to check. It happens. Now I know where to look. Do I think there will be another high severity vulnerability in one of my Django projects? It's possible, but if I put a bit of time into it, it may be the one.
Read more »
Jan 17, 2013
I'm in Brasil now. I already posted a blog on my Brasil blog.
Short story time. Twitter doesn't work in the subway and I didn't want to write interesting stuff to Twitter after it ate a message, so here are a few notes about Tuesday night in New York.
Read more »
Jan 12, 2013
I've been blogging more than usual since I released AI3 on Christmas Eve. You should check it out. In comparison to all websites I have released, AI3 has the most potential and should get the most respect. I purchased a super-fast server (SSD especially for fast database lookups), leased a super-fast colo space for it, and am going to add to it regularly. As a feature of AI3, I will attempt to keep a regular blog here with insight into what I think about each feature of the website is and then I will make a page with that data on ai3 using a simple slug. I've already done a few if you want to look at the past few blog posts.
The feature that I'm going to discuss today is single-minded research of a single difficult topic. Searching for a common word in Google can be one of the most frustrating things in the world. What you really want is for someone to answer the question you are asking, not learn every way to misunderstand what you are asking. Sometimes AI3 will fail, there's no doubt that Google is more in depth than anything I can create even if I had all of Wikipedia. So let's get in depth on a very simple question. It's not one of the easy questions I've been dealing with. Let's ask: "Is the word 'We' used more positively or negatively?" By that, I mean "Is the sentence 'We plan to solve poverty by 2017,' more common than 'We can not solve poverty by 2017'?" But not just that sentence, but every sentence which is in the positive "We *verb*" vs "We *verb* not". This is a deviously difficult problem. Even with a huge corpus, definitive answers require statistical analysis of a ton of stuff. Let's attempt it though. Start with We and we. All words in AI3 are case-sensitive, which is why there are links to all variants of we on the We word page. 1276 pages is too many unless we have a script. Let's try collocation of We. It's a slow process because We is such a common word. You can look below if you're impatient. While you're waiting, maybe try looking at a few sentences. The second sentence is:
`` We didn't want town work '', Jones said.Eureka already? Yup. All we need to do is find similar words on We and every word that is in the negative. That's pretty easy, right? There are only four pages of words that contain n't and most of them are pretty uncommon. Note that there's a bug where dashes assume that two words are one. That's a problem with my parser which should be more intelligent about whitespace. So manually or automatically, we can start searching for sentences that contain We didn't and so on. Since the related page doesn't have a count (due to slowness), we are stuck just trying a high page number and using a binary search from there. If you don't know what a binary search is, let me explain. Let's say that there could be upwards of 100 pages of sentences or more. Simply skip to page 100. If it gives you an error, then there aren't that many pages. Go to half that number, page 50. Half the number again and again until you come up a valid page. Then pick a number half way between the valid page and the invalid page. After a few hits, you will find that page 6 is the end of We didn't. In total, it should only take 7 tries to find any number between 1 and 100 because 2^7 is 128. If you don't understand the math, hopefully you'll understand the process. Anyway, now we have a way of counting all the negative sentences. Then we simply need to count all the sentences that contain We. That can be found on the We word page. But let's say that you thought this algorithm through and have some skill with a database. How long would it take you to come up with the solution? Read more »
Jan 2, 2013
AltSci will soon have a proper server with a lot of bandwidth. I just bought a $1000 server off Newegg and it's going to be fast. Intel Xeon E3-1230 V2 Ivy Bridge 3.3GHz (3.7GHz Turbo) 8 threads, 16 GB ECC RAM, 240GB SSD. I'm going to send it to a datacenter with a lot of bandwidth to do some fun projects.
As I spend less and less time working on my new website AI3, I have to choose the most important features that I want to work on. Completeness is important (the page on Emma Goldman is missing everything after the header), social features are important (users aren't implemented yet), UI is important, and adding more data sources is important to my users (Creative Commons blogs will be added as time goes on), but more important to me is functionality.
Read more »
