MongoDB tips from "Lessons Learned from Migrating 2+ Billion Documents at Craigslist"

You should really listen to this talk "Lessons Learned from Migrating 2+ Billion Documents at Craigslist" by Jeremy Zawodny.
However if you don't have 30 minutes to spare these are the main items:

1. Pay attention to encoding. MongoDB uses UTF8 so you'll need to process your data if it has all sorts of encoding.
2.  There's a document size limit (defferrs from version to version) so if some of your documents are too big you should plan how to avoid this problem. Otherwise it will fail when you'll try to load them into MongoDB.
3. Pay attention to data types (don't put everything as string) - otherwise you'll have trouble when querying. This is especially tricky when using dynamic typed programming languages. Also make sure that the driver you use is not inferring your data types.
4. Sharding - when you first load the data you can  stop the internal load balancer (to reduce IO) and you can also  pre split the data in advance.
5. consider using file system that supports compression if you store lots of text.

and finally - join the mailing list. it has tons of information that would be very helpful.

rails with sqlite - suddenly became a little nightmare

So I decided to reinstall rails on one of my virtual machines since I really fucked it up and there was no restore image available. Though to myself I'll just waste 3 minutes and everything will be up and running. What I did not expect is that reinstalling sqlite3 which I use for Dev will cause me such an headache.

while creating new project using "rails new someProject" the bundler had problems with the sqlite installation saying "an error occurred while installing sqlite3 (1.3.6) " . Googled a bit about it and found so many people having the same problem but nothing worked. the gem file and the sqlite3 installation did not want to work together. The solutions varied from tweaking every possible file related to ruby to uninstalling everything you ever had....
after trying all sorts of voodoo from  stackoverflow I found out the solution (among 10 other I tried) which is to install a dev version of sqlite using "apt-get install libsqlite3-dev".
So first of all I posted it here not to forget it once I'll run into this weird issue again. What surprised me the most is that nothing in the error messages implied that the problem is with the version itself nor did the gem file gave me any clue about it.

Getting back onto linux after so many years working only with windows surely takes its toll.

Two issues with asp.net MVC

While Microsoft pushes it's MVC implementation over the old web-forms model it seems that some of the techniques demonstrated in most presentations/introduction are quite naive and can lead a developers problematic paths. Here are a two things that might look small and not very harmful but I think they should be taken in consideration when using Asp.net MVC:


1. Security issue with the default model binding - The implementation of the default model binding takes the data coming from the users HTTP request and puts it in a key-value data structure. This is a good thing for saving ourselves from boilerplate code but using it without thinking of security implications can lead to some security flaws in the code. For example,  with a little common-sense/internal-datastruct-knowledge/fuzzing-knowledge an attacker can alternate internal values of the model (or view-model, depends on the programmer) by adding them to the http response sent from his browser to the server. This can lead to all sorts of bad things from impersonation to data corruption and even perssisted attacks. it all depends on the flow of the code. you can checkout this simple example


2. I've seen excessive use of RedirectToAction fundtion which causes HTTP 302 which google is not very fond of. HTTP 302 means that the page has temporarily moved to a new location. this, however, is not what is usually demonstrated. it is usually used in Asp.net MVC to demonstrate the way to redirect a user to another page after performing some action. a better approach would be to use some kind of Server.transfer but it is missing from the current MVC implementation. You can check the workaround but I think such function should have been implemented inside the framework itself.