PhotoRec just saved my day
The file system on my SD card got thrashed and I couldn't get my pictures the normal way anymore. PhotoRec, a tool of thetestdisk package, could get everything back! Including previously deleted pictures...Python attribute caching with low overhead
For my work I need to do various lengthy numerical calculations on large datasets with results that do not change. The program is interactive, which means I do not know in advance which calculations on which data I need to perform. The results of the calculations depend completely on the dataset, there are no additional parameters. One can implement such a thing with a cache and corresponding code that checks if the result is in the cache before doing the calculation. I developed a generic technique to cache the results in such a way that no function call is necessary at all if the result is cached. It works by dynamically generating and deleting object attributes. If the attribute is set, it is used. If it is not set, a corresponding function to calculate the attribute is called and the attribute is set. If the data changes, all designated attributes are unset and will be recalculated when they are accessed next time.
A class for lists demonstrates how it works:
#!/usr/bin/env python3
# works similarly with python 2.x, except different syntax in some points
# -*- coding: utf-8 -*-
class CachedList(object):
# List of attributes that can be cached.
# Use keys of a hash to quickly look up if an attribute is managed.
# Leading underscores do not work due to the way attributes
# with two underscores are handled.
# A method that calculates the result has to be defined for each of these attributes.
# It has the same name as the attribute plus a leading underscore: average -> _average(), ...
_managed_attributes = {'average': 1, 'sum': 1, 'len': 1, 'exc': 1}
def __init__(self, l):
# copy the list
self._list = list(l)
self._exception_cache = {}
# This method is called automatically if an undefined attribute is accessed.
# If the attribute already exists, this method will not be called.
def __getattr__(self, attr):
# Avoid recalculating attributes that raised an exception.
# Instead raise the cached exception.
# Some of my calculations fail with an exception after a very long calculation,
# so it makes sense to cache the exception.
exception = self._exception_cache.get(attr)
if exception is not None:
raise exception
if attr in self._managed_attributes:
# Try to get and call a method that "makes" the entry
f = getattr(self, '_'+attr)
try:
# Just some tracing code for the example
print("calculate {0}".format(attr))
result = f()
setattr(self, attr, result)
except Exception as e:
# "remember" the exception
self._exception_cache[attr] = e
raise
else:
raise AttributeError('{0} object has no attribute {1}'.format(type(self), attr))
# result has been set if this point is reached
return result
def append(self, x):
self._list.append(x)
self._invalidate_cache()
def _invalidate_cache(self):
# just some tracing
print("empty cache")
self._exception_cache = {}
for a in self._managed_attributes.keys():
try:
delattr(self, a)
except AttributeError:
pass
def _average(self):
# Cached attributes can be used just like normal attributes
return self.sum/self.len
def _sum(self):
return sum(self._list)
def _len(self):
return len(self._list)
def _exc(self):
raise Exception("Exception!")
# now test the code!
l = range(10000000)
cl = CachedList(l)
# attribute is calculated and set
print(cl.average)
# attribute is used
print(cl.average)
# cache is cleared
cl.append(100000000)
# attribute is recalculated
print(cl.average)
# exceptions are also cached
try:
print(cl.exc)
except Exception as e:
print(e)
pass
try:
print(cl.exc)
except Exception as e:
print(e)
pass
user@host:~$ ./propertycache.py calculate average calculate sum calculate len 4999999.5 4999999.5 empty cache calculate average calculate sum calculate len 5000008.9999991 calculate exc Exception! Exception!
Why is the air in airplanes so dry?
If you ever traveled a long distance by plane, you may have noticed that the air is extremely dry: The eyes start burning, the skin is dry and you get thirsty. But why is the air not moistened so that the passengers are more comfortable?
There are actually two reasons. First, the air at the altitudes where the plane is flying is extremely cold (say, -60 °C and below) and consequently dry. It contains practically no water. This air is compressed by the turbines and used as cabin air. Any water in the air would come from people breathing or would have to be added. Such a "moisturizing" system costs money, takes up space and has weight, so an airline would have to think if the added cost is worth it.
Second, airplanes are usually made of high-strength aluminum alloys that can corrode in contact with water. They also have a lot of electrical cables running all over. And they fly at high altitude where it is very, very cold. So at least some parts of the hull are cold, too. This means that moisture from the air inside the plane could condense to liquid water on cold parts and cause trouble. Dry cabin air can reliably prevent that: If the dew point (resp. frost point in that case) of the cabin air is below the freezing point of water, there can be no liquid water condensing. Only frost can form, i.e. water vapor turns directly into ice, which does not cause corrosion problems. And a frost point of roughly 0 °C corresponds to a relative humidity of 25 % at 20 °C. If the humidity in the cabin is desired to be above 25 %, one would have to take special measures: Corrosion-resistant materials or a completely sealed vapor barrier on top of the cabin's thermal insulation. Such a barrier would prevent the moisture from reaching colder parts. But it would make the insulation more complex and add weight...
Libreoffice WTF
Libreoffice bugged out on me big time (3.5.3.2 350m1(Build:2)). A presentation that I created at my work computer has mangled figures if I open it on my home computer. On exactly the same version of libreoffice! Things breaking when I open a file with a different version or when I have to export as .ppt I can tolerate. But incompatibility between different installations of the same version is beyond my comprehension and strains my patience. Maybe I should give LaTeX Beamer another look.
Does MS Powerpoint still do this kind of stuff?
Currency backed by gold?
In these economically volatile times with the 2009 financial crisis and the European debt crisis, some people argue that a gold-based currency would be desirable because gold is a "real value", as opposed to pieces of paper with elaborate imprints on them or numbers in a database. But how is gold a value more real than paper money or bank accounts? As such, a piece of gold seems rather useless to me. It can serve as paperweight, blunt weapon or door stopper. I admit that you can make pretty, shiny things out of it, but, lets say, martensitic stainless steel in the shape of a knife is more useful in the daily life. Should we have a currency backed by martensitic stainless steel?
The value of gold, just like paper money, is based on the fact that other people want to have it and that they are willing to trade their work force or posessions for it. The demand for it has always been higher than the production capacity, and mining gold is an expensive and slow process in itself. It is therefore quite suitable as a currency in a primitive society: It's supply is self-limiting and keeps the value of gold at a more or less constant level that is related to it's mining cost. A rising gold price makes gold mining more profitable and increases supply, while a falling gold price discourages mining.
But on the long run and in a growing global economy, a gold-based currency could be fatal for the following reasons:
- Gold mining and refining is bad for the environment.
- Mining gold that is then stored in some safe and "sitting around" wastes all the expenses that went into mining and refining it, because the gold is not doing anything useful. The gold should be used as jewellery or for high-tech products.
- The mining cost of gold is not constant. It could decrease because new technology is developed that decreases the minig cost, or new deposits or production methods are discovered. It could increase if existing deposits are exhausted. Both could disrupt the value of gold.
- The production of gold cannot be increased quickly, which could lead to a deflation if the world economy outgrows the gold production.
- The value of the gold currency would not be under control of central banks, but under the control of mining companies. They would in fact be the new central banks.
- Gold is not distributed evenly around the world. It could create a similar dependency like for fossile fuels.
- Gold is at the moment and will be in the future subject to speculation. It's value is more volatile than good currencies like US-Dollar or Euro. The value of such currencies is under tight control of powerful and highly developed central banks that stabilize them proactively.
There is a good reason for which gold was abandoned as a currency or base of a currency. Gold is not a real value. Real values are the house that you live in, the food that you eat, the tools that you use, the things that you know, and most importantly, the people who you love and who love you. All these don't make a good currency, but definitely a good investment.
What is Greece doing?
Just now the Greek prime minister Georgios Papandreou announced that he wants to conduct a referendum if Greece is going to accept the EU financial aid. This came as a surprise to everybody, as just last week the European countries came to a first agreement on the financial aids after a long struggle. The markets are not so fond of Papandreou's move and indices are plummeting as I am writing this. Respecting the will of your population is not a bad thing in itself, but a referendum in such a heated environment with street protests all over in Greece resembles tossing a coin instead of making a rational decision. Papandreou's government will have to inform it's citizens about the implications of their decision, and I hope people will think and decide carefully. My feeling is that the Greek government wants to dodge the responsibility for their bad decisions in the past and later blame the population for whatever decision they take in the referendum.
The options, as I see it, are harsh austerity measures on all fronts in case Greece accepts the aid and stays in the euro zone, or the revival of the drachm and bankruptcy. Both options are at first viable ways out of the situation, and I'll try to analyse what could happen in either case, maybe identifying a preferable solution.
Staying in the zone
Four things will be necessary to get out of debt: A debt cut as an incentive to pay back the debt at least partially instead of going bankrupt right away, reducing government spending, increasing government income and improving the efficiency of the Greek economy in order to keep an acceptable standard of living for the population. The debt cut is already agreed on, but the remaining three steps are politically problematic. The main hindrance is a deep-rooted culture of corruption, tax evasion and clientele politics in Greece: Decisions are not made based on what would be better for the country, but what would be better for the decision maker and his friends. Once such a system is fully established, it is hard to remove because the network of corruption includes judges, prosecutors and high government officials who would be the first in line to fight corruption and tax evasion. They will be preoccupied with saving themselves instead of fighting the rot in their own ranks.
The only way here is to fight the corruption from top to bottom. This means that Papandreou will first have to fight corruption in his own government, which may sever him from a lot of important figures that support him. Does he have enough people around him who are loyal to Greece? If not, there's no hope and Papandreou will fall soon.
Fighting corruption, reducing government spending and increasing government income have to be backed up with stimulating the Greek economy in order to fight unemployment and keep an acceptable standard of living for the population. Only the private sector can create income from which to draw taxes and pay salaries to workers. Greece has to attract foreign capital, support export-oriented industries to fight the foreign trade deficit, educate people and train them for jobs that are needed, and also invest into infrastructure — which is difficult if you up to your neck in debt. Again, improving government efficiency is a key factor because slow and unreliable bureaucracy hand in hand with corruption repel investors and make life hard for domestic companies.
Leaving the zone
It would mean that foreign investors can pretty much kiss their money goodbye. Probably (I'm just guessing here) the foreign debts will be converted to drachm at a rate dictated by the Greek government, and the expected gradual depreciation of the drachm with respect to dollar and euro will devalue the debts. This step would therefore be strongly against the interest of any country or institution that lent money to Greece or Greek companies. For me it is still a question how the debts of Greek companies, as opposed to government debts, would be handled.
The diminishing buying power of the drachm can be a chance for Greece: The country becomes more attractive for investors because of lower levels of income, and export-oriented industries and tourism are bolstered by a weak drachm. Problem here is the EU membership, which pretty much forbids tariffs for trade within the EU. It means that the price of many goods in drachm will increase as the exchange rate of the drachm drops relative to the euro. Similar things have happened in north America, where international trade has increased the price of corn in Mexico, hurting the Mexican population.
On top of that, Greece still as a foreign trade deficit, which means that the effect of increased price of imported foreign goods outweighs the benefit for the own export-oriented industries. And, again, it is the private sector that ultimately creates value in an economy, and a strong private sector needs an efficient and reliable administration as well as good infrastructure and educated workers.
The reduced international debts will only provide a short-term relief if the deficit remains: Taking new foreign debts will be much more expensive, and "printing drachms" just lets the value of the drachm spiral down into a deep inflation.
Conclusion
In summary, reintroducing the drachm is not going to solve any problems. Such a step may be beneficial in some situations, but in case of Greece with it's foreign trade deficit and continuing accumulation of debts it only makes the situation worse. The way out of the crisis is the same in any case: Fighting corruption and tax evasion, making the administration more efficient, and stimulating the economy. Money for military and "presents" to diverse interest groups should be diverted into education and infrastructure. And Greece has to become a reliable partner for foreign investors and other EU countries to ensure support during this time of crisis.
Papandreou's move is therefore clearly in his own interest by putting the responsibility for any kind of measures to the voters and not to him and his government. He also seeks to win time, probably trying to save him and his buddies from the bloodbath in the awaiting fight against corruption and tax evasion. But at the same time he's gambling with his country's future, as leaving the euro zone is definitely not a good option for Greece. Instead, he is hurting the relations to other EU countries and also private investors, destroying any trust into the willingness of the current administration to take the necessary measures.
The debts to Greece are lost at least partially in any case, and a debt cut along with financial aid and political pressure to solve the domestic problems are probably the best way to recover at least some of the loans to Greece and Greek companies. The damage to other EU countries and companies could be severe on the short run if Greece reintroduces the drachm because of writing off the debts of Greece, but the strong economic position of other EU countries would soon allow them to recover. If other countries like Italy, Portugal, Spain or Ireland however follow the path of Greece it would severely weaken the EU. I believe that sticking together and helping out each other is in the interest of all European countries.
Numpy makes your python math 10x faster
As an exercise in parallel number crunching with python and numpy, I wrote a program for the computer language benchmark game. It is more than 10x faster than the python runner-up without numpy and plays already in the league of compiled languages. The speedup comes solely from the high-level vector math functions of numpy.
As a remark: The program was disqualified from the competition but is shown anyway. I don't exactly understand why. Well, I also don't really feel like asking the admin Isaac Gouy, as he seems to be a bit tense about the subject.
Two-column float gets stuck with revtex
When preparing a manuscript with revtex (documentclass[aip,apl,amsmath,amssymb,reprint]{revtex4-1}), I ran into a problem: A two-column float with the figure* environment got stuck and ended up at the end of the document. The solution was trivial: Remove all placing hints for the float, instead of \begin{figure*}[h] use just \begin{figure*}!C and C++ vs. interpreted languages like Python
Check out these articles!
- A Guide to Undefined Behavior in C and C++, Part 1, Part 2, Part 3
- What Every C Programmer Should Know About Undefined Behavior, Part 1, Part 2, Part 3
A high-level summary: The C and C++ standards are defined in such a way that the result of many operations is undefined if it would require additional checks to make it defined on all platforms. This includes dereferencing invalid pointers, signed integer overflows and shifting beyond bit boundaries. The compiler has lots of room to rearrange or drop code which consequently leads to a fast program. On the down side, it can lead to unexpected and unsafe behavior in many cases. Informing the user about all cases where the code could possibly be unsafe would be either too noisy with tons of false positives or contain too many false negatives.
It seem that such errors are kind of unavoidable in large codebases because only geniuses can figure out what exactly is going on and the bytecode that optimizing compilers generate has little resemblance of the corresponding source code because of loop unrolling, inlining and so on.
A language like Python looks to me like a very good solution: The programming language has many high-level functions and libraries with completely defined behavior, and the average programmer does not have to worry about undefined behavior. These high-level functions are implemented as highly efficient and thoroughly tested C routines. The interpreter will then, in an ideal case, spend a significant amount of time in these optimized functions (say, for example, multiplying numpy matrices) which reduces the overhead from permanent bounds checking and so on. For the interpreted language it is critical that moving parts of the code to C is easy, for example by allowing calls to libraries (example: ctypes) or implementing subroutines in C (example). C is in those cases only used to speed up the performance bottlenecks or access existing routines.
Another layer of abstraction
Apparently Microsoft wants people to write applications for Windows 8 in HTML and JavaScript. And many people like that idea. Now I am wondering how that makes sense. I already don't understand why Android, for example, uses Java instead of native code for applications, because especially on smartphones it would be important to get as much as possible out of the scarce hardware resources, right?
Well, I am using Python instead of C++ for my own applications, although they really need a lot of computing power. I believe it makes sense because I am using Python mainly to glue together a bunch of different libraries which are often written in a compiled language, so that the really speed-sensitive parts run as native code, like drawing the UI and doing hardcore math. Having less headache during development and more compact code does hopefully compensate for the speed penalty of the program, but only because I am the only user of it so that the development time has the same weight as the waiting time when using the program.
But how does that figure for applications used by many users, like for example text processing? MS Office and LibreOffice feel quite sluggish to me, especially when loading huge documents with big images. Is that because of I/O or because of processor cycles being used inside the program? What about PDF readers and the like? One should see how much processor cycles are actually used by, say, a mail client compared to an AJAX webmail interface running in a browser. Maybe it even makes sense to write a text processor in HTML + JavaScript, because browsers are pretty good at drawing text, images and lately vector graphics on the screen, so perhaps even UI libraries like QT, GTK and the likes are kind of obsolete.
What do you think? Do browsers make a fair runtime environment for desktop applications or even more specialized software?
