Gametop.com Games without Gametop.com Redirection

I like this game library site, there are dozen HQ free racing (my favorite genre) games  for download. As we know the games upon exiting will launch your browser pointing to their site. There seems to be no catch here, no suspicious parameter etc. it's simply "open this url" call, although a googling for privacy concern of these games return a few security paranoid's comments.

Having a look from dependency walker I believe ShellExecute is responsible for the call. I try hex it with another same-length function name of shell32.dll (e.g. CheckEscapes) and yeah that's it.

How to build PyGI AIO with Moluccas

EDIT: This guide is WIP do not follow! I will revise and resume it when final pygi aio binary done

This is the first part of possibly long guide to build PyGObject aka PyGI for Windows using open source compiler GCC. Several users has been asked how to build it but since I was confused as well how to do it (let alone explain it), thus I never wrote one. So here you go (there still lots of gotchas)!

Read about Moluccas first, its gcc has experimental CRT targeting for this guide.

MS CRT targeting initially laid by official mingw.org but still far incomplete. At first I thought it would be as easy as set cflag/cxxflag __MSVCRT_VERSION__=0x0900 and add -lmsvcr90 to GCC's specs file, well for a trivial program? yes it can but thats all it can get. As my experiment goes by there are many issues piling up (from incomplete msvcr implib to puzzled LFS/Y3K macros) that forcing me to modify mingw's crt and win32api files to allow a package completely linked to the new msvcrt dll. In the end I decided to add new GCC flags to simplify this: -vcr70 to -vcr110. The flags correspond msvcrt redist version bundled by each release of Visual C++. Even today it still far from complete.

What it has to do with this guide? there is an article wrote by win32 contributors of Gnome framework (or whatever you called it) that explain the concern. While my approach is far cry in comparison to use real thing (yeah I know MSVC has free express edition) it still useful in some cases at least for OSS zealot (even if they use Windoze).

This guide will focus on latest stable version of GTK-related (should be called GObject-related) libraries as this written. The main packages are Glib 2.38.1 + GTK 3.10.2 + GI 1.38.0 + PyGObject 3.10.1 and we are targeting python 3.3 (which use msvcr100.dll). I must tell you though, there is steady progress to support MSVC from upstream at least for major libraries including GI thingy.

If you don't need CRT targeting, you can use any GCC of your preference in moluccas by put it into moluccas root then switch over using setgcc [gcc folder name].

Download moluccas 0.8a, you need to download the following packages:

M08.exe
M08-basemsys.7z
M08-basetools.7z
M08-ccomp.7z
M08-tex.7z

1. Saves all of them in the same folder,
2. Run M08.exe and specify install path (it's a portable installation). Click OK. Correction: If you have conemu instance(s) already run you need to quit all before clicking OK, see Maximus comment.
3. The installer dialog will vanish quickly to bootstrap conemu and continue with extraction. Due to the sheer number of files, the extraction (write to harddisk) will be way slower than what the CPU capable of.
4. Once completed, run conemu.exe to start
5. Moluccas will try to detect any presence of Visual C++ and try to integrate the latest version you have. During first run it also initialize Miktex installation
6. In the end conemu panel will bring the <^> prompt
7. Type spawn to start an instance of conemu. The panel will hide and still accessible via systray icon.

Download sources:
pixman-0.30.2.tar.gz
cairo-1.12.16.tar.xz
glib-2.38.1.tar.xz
pango-1.36.0.tar.xz
atk-2.10.0.tar.xz
gdk-pixbuf-2.30.0.tar.xz
gtk+-3.10.2.tar.xz
gobject-introspection-1.38.0.tar.xz
pygobject-3.10.1.tar.xz
freetype-2.5.0.1.tar.bz2
dbus-1.6.16.tar.gz
fontconfig-2.11.0.tar.bz2
harfbuzz-0.9.22.tar.bz2
libjpeg-turbo-1.3.0.tar.gz
gettext-0.18.3.1.tar.gz
libpng-1.6.6.tar.xz
zlib-1.2.8.tar.xz
libxml2-2.9.1.tar.gz
graphite2-1.2.3.tgz
libffi-3.0.13.tar.gz

You may use wget and tar to extract or doing it with your browser and 7zip. Just be sure to extract them somewhere in non spaced path name.

In moluccas there is 5 different GCC, each has specific purpose: gcc3, mingw32, mingw-w64, gcc64 and msysdev. Those are just folder name where each reside (not "actual" name), the default is mingw32 (gcc 4.7.3 based on mingw.org) which is still de-facto standard (meaning: you get less trouble). Gcc3 (gcc 3.4.6) is the oldest, msysdev (gcc 3.4.4) is for making msys binary, mingw-w64 is a 32/64 multilib gcc 4.7.3 based on ming-w64 project and lastly gcc64 is the new gcc 4.8.1 targeting 64bit only and has SEH support. For now on when I say mingw32 or gcc3 I really mean the active gcc used in moluccas.

Before start compiling we need to prepare compiler flags. Moluccas gcc 4.7 has internal flags: -march=pentium-m -mfpmath=sse or roughly say "turned on all intrinsic optimizations up to SSE2 and use SSE2 for math too". But I found SSE2 sometime lead undesirable result, in this case the C based apps appear doing fine but in python mode it silently crashed. So just to be safe we will detune it a bit by adding -mno-sse2. Moluccas also has default cflags/cxxflags as -Os and ldflags -Wl,-s this is just my preference, feel free to modify it. For release mode I would add another flags -fno-unwind-tables -fno-asynchronous-unwind-tables which are specific to mingw32 to strip further some features of dwarf-2. For CRT targeting use -vcr100 in both compile and linking flags, in accordance to MSVC this means use msvcr100.dll, drop some POSIX named functions (yeah damn MS), turn on Y3K but not LFS. But I want LFS here so to turned it on add -D_FILE_OFFSET_BITS=64. Should match with how python built

export CFLAGS="-vcr100 -D__MSVCRT_VERSION__=0x0900 -D__NO_MINGW_LFS -Os -fno-unwind-tables -fno-asynchronous-unwind-tables  -mno-sse2"
export CXXFLAGS="-vcr100 -D__MSVCRT_VERSION__=0x0900 -D__NO_MINGW_LFS  -Os -fno-unwind-tables -fno-asynchronous-unwind-tables  -mno-sse2" 
export LDFLAGS="-L/local/lib -vcr100 -Wl,-s"
export CPPFLAGS="-I/local/include"
export PKG_CONFIG_PATH="/local/lib/pkgconfig"

The gotchas:

As we already know MSVCRT 8 and 9 (for targeting python 2.7) use side by side installation (those piling up stuff in %WINDIR%\WinSxS) but not with version 10 and later. Those SxS need an executable to be manifested to "find" the correct version of the dll. In reality though, it will use the latest installed redist. Non manifested will throw error, moluccas' gcc 4.7 will do this job by append precompiled manifest file before sending it to ld (linker). However if executable already use resource (.rc) file, the result there will be two resources at the end of PE with only the first laid recognized (the manifest discarded) which not what I want :-(. So if you have Windows SDK detected by moluccas there is mt8/mt9 command to clear the issue. This is undesired because during compile sometime the newly built exe immediately used to do something.

The next gotcha is libtool and C++, by default in a C++ project libtool will override g++ behavior by enabling -nostdlib and reconstructed the objects files and PE stubs *including* the linker flags "the way libtool want it to be". Thus break the purpose of -vcrXX (actually it strip the flag out carelessly), it even strip valid linker flag such -pthread. I don't know if there a way to override this behavior because fixing it from configure script is quite PITA.

When used with mingw32 flags that trigger __STRICT_ANSI__ will break CRT targeting because those underscore prefixed stuff being disabled. So -std=blablabla flags should be disabled

Another gotcha is LFS and Y3K (64bit file offset and time), LFS:off and Y3K:on is how official 32bit python build, LFS:off and Y3K:off is the how official GLib (see glib\gstdio.c) build for windows 32bit. And here I opt to turn all on. Hence it's ABI breaking if we try to mixed Glib' dlls.

Let's start with non gobject related ones. When I say "gobject related" it mean those that depend on Glib, others like cairo and harfbuzz are borderline, they are independent but can be configured to support gobject.

For make install there is the usual /usr/local which by the default has higher priority in $PATH environment than /usr/bin (msys realm). In this guide it's bad because some of packages will install executable that clash with msys' binaries. I would recommend create different folder and append it to $PATH instead, but for now lets just rename those conflicting files.


non gobject related:

zlib
make -f win32/Makefile.gcc -j2 install SHARED_MODE=1 INCLUDE_PATH=/local/include BINARY_PATH=/local/bin LIBRARY_PATH=/local/lib

libffi
configure --disable-static --enable-portable-binary
# it's important to add  -enable-portable-binary  otherwise it will use native cpu optimization
make -j2 && make install

gettext
patches:

cd gettext-runtime
# actually we just need the tiny intl -.-' what a waste of time of configure
configure --with-included-libxml --without-emacs --with-included-libcroco --with-included-libunistring --with-included-glib --with-included-gettext --disable-static --disable-java --disable-csharp --enable-threads=win32
cd intl
make -j2 && make install


libpng
configure --disable-static
make -j2 && make install

pixman
configure --disable-shared
make -j2 && make install


libxml2
configure --disable-static --without-python
make -j2 && make install
# do not interfere msys
mv /local/bin/xmlcatalog.exe /local/bin/xmlcatalog.bak


freetype
configure --without-bzip2 --disable-static --prefix=/local
make -j2 && make install

fontconfig
patches:
fontconfig-2.11.0_binlibpath.patch
fontconfig-2.11.0_notest.patch
configure --disable-static --enable-iconv --enable-libxml2 --disable-docs LIBS=-lregex
make -j2 && make install



gobject-related libraries:

glib
patches:
glib-2.38.1_clockgettime_vista.patch
glib-2.38.1_staticbuild.patch
glib-2.38.1_FDnewCRT.patch
# newer glib will try to detect pthread presence, lets specify what we want
# glib spawn also not work under newer CRT to compile need to switch to mingw-w64
configure --disable-static --with-threads=win32
make -j2
cd glib && setgcc mingw-w64 && rm *.exe && make -j2
cd .. && setgcc mingw32 && make install


cairo
# we planned to try accelerated composition/webgl for webkitgtk later so lets try enable cairo-gl
# strangely egl need specifically disabled here
patches:
configure --disable-static --enable-gl --enable-wgl --disable-egl 

# cairo-gl tests will fail to compile due to glx, lets force it
make -j2 -k
make -k install


Now things get hairy... 

Last year I attempted to port python 2.6.7 to msys. Basically I saw roadblocks ahead... not only msysdev is too old to do modern thing but it was born as son of a sorry-state cygwin's gcc. Sure it will be way easier to port GI with msys-python (less maintenance at upstream) but then if we have msys-python what about scons and dozen buildtools out there? still lot of patches and maintenance involved!. Moluccas has msys-python that switchable with native one (both at the same version) but we are not going to use it, we use Dieter patch instead with some workarounds from me.


What GI does? well, it "introspect" the sources files of gobject based library and generate "binding" (more like a translation dictionary IMO) in xml form of gir files and compile (also strip docs) into binary typelib files. When python script call a function GI will look up these dictionaries how that function will call the C equivalent. This way it's not a strict binding like the usual pyd files so if typelib contain an incomplete references to a dll library we wont get error until we call that missing function, vice versa. That is older typelib may appear usable for newer dll and vice versa, assumed a compatibility is warranted. These typelibs collected under lib\girepository-1.0 subdirectory and its basename (excluding version) become the hint how to import it in python.

During binding generation GI create a dumper in temporary folder. This dumper has problems with mingw-w64 headers and its PE stubs (often the exe wont execute at all)  but seems to run fine when mingw32 used.

gobject-introspection
patches:
gobject-introspection-1.38.0_mingw_msys.patch
cp /opt/python/python-config /usr/bin/python-config
configure --with-cairo --disable-static CPPFLAGS="-I/local/include -I/opt/python/include"

# broken with -j2
make && make install

If above compile fine, we will have some gir problems: redefined win32 functions. When transitioned to gtk3 some libraries such gdk, pango, atk and glib remain compatible with both gtk2 and gtk3. At least gdk-pixbuf has several function redefined with *_utf8 suffix for win32 via C directive. They should be cross platform under python so the python call should renamed back while referring to *_utf8 C functions. Maybe these fixes can be implemented at pygobject's overrides but I haven't try though.

As with the latest GI, there are less error in the gir generation, but docstrings are often missing.


Python related

py3cairo

TBC

Moluccas bundled python is 2.6.7, PyGObjec 3.10 drop 2.6 support. In this case we need to produce two files to ease compiling:
/bin/python2 for python 2.7 (its shebang refer to C:\python27\python, see /bin/python for reference)
/bin/python2-config for python 2.7 (see /opt/python/python-config for reference)
/bin/python3 for python 3.3
/bin/python3-config for python 3.3

pygobject
patches:

pygobject-3.10.1_noansi.patch

cp /c/python33/libs/libpython33.a /local/lib/libpython3.3.a
configure --enable-cairo PYTHON=python3 CPPFLAGS="-I/local/include -I/c/python33/include"
# the tests forgot -no-undefined libtool flag lets move on
make -j2 -k 
# install will be incorrect (pyd not copied), need to fix the makefile
make -k install

PyGI 3.10 Windows

In my attempt to update MyPaint I need to update PyGI as well as GTK. During compiling there is worrying result the glib 2.38.1 failed on 7 tests! The tests a bit different from previous versions though (in he past usually it passed all tests). GTK 3.10.2 seems missing def files and the new one isn't look any stabler than 3.6, the usual crashes still there despite major win32 fixes in cairo 1.12.16. Lastly I can't use new bindings (gir/typelib) as it always throw gmem.c overflow error during initialize in addition to that I need to comment PollFD override chunk in gi\overrides\GLib.py. Anyhow the new generated girs will still need diff/merge from those linux version...

So here is the new binaries using older typelibs (from pygi aio 3.4.2). This means no new features at python level, plus bugfixes and probably new bugs in the runtime. Currently only bare gtk included if new typelibs become usable I will add others.

pygi-aio-3.10_win32.7z (python 2.6 unsupported)

EDIT: rev2 and further revisions goes here https://sourceforge.net/p/pygobjectwin32/files

Resuming Download from Depositfiles

Don't know if someone has wrote this before (cause it's so simple). This weekend I desperately need several downloads from one of major filesharing survivor DepositFiles. The files are big and my ISP is slow (worse with shared IP). Don't want to mess with eternal redownload or mess around with onion network, I'm looking for a way to resume the download (without their downloader of course).

I found at least two ways: First to use free URL uploader service which offer resumable download or secondly using wget (or whatever downloader with resume ability).

We will need to get Depositfiles' actual link to do it, as we know the url is in this kind of form http://fileshare*depositfiles* we can use source view or built-in browser inspector to get it. Of course we should never click the download button.

Free URL uploader is probably quite risky business, I have seen this kind service very hard to survive in the past. This time I found http://d-h.st/, the site seems clean and modest but not without problem. As expected, the download is very unstable, keep broken every 1 MB. Don't worry wget will deal with it. Interestingly registration appear to be optional. Just upload and the resumable download link will be provided.

The second way need tool to kill specific TCP connection ala task manager or taskkill, I'm not aware of the availability of the tools under Windows out of box but 3rd party tools such Sysinternals TCPView or ProcessHacker will do.

Essentially what happened is when wget issue resume command the server will get "stuck" but only initially, this is when we need to close wget tcp connection. FYI, closing tcp connection is treated as disconnection from the server so wget will do retry instead of giving up. Upon retrying the "stuck" is "unlocked" and download become resumable.

Enjoy!

Weeks of Maps and TPX

I've been spending 2 weeks dealing with maps for several related projects, exploring several (new to me) better projections such Cahill-Keyes-Waterman and Fuller. Probably GIS guys already knew this stuff but in case anyone (newly) interested in maps I have some tips.

OK first what about google maps, google earth or wikimapia? well if you ever find a reason for not using online resources then read on.

Since many Earth datasets has fall under public use, there are many efforts to bring "open source" friendly yet highly detailed maps for mere mortals. Unless you're GIS guys you may not need SRTM, Landsat, VMAP, WVS, CDC's shapefiles and others as there are several ready to use (or easy to setup) maps for people with zero GIS experience.

Vector maps:
- Wikipedia users have made accessible vector maps here (the most detailed svg): http://commons.wikimedia.org/wiki/File:World98%2B.svg
- Duncan Webb's Cahill-Keyes World Map (draft) : http://www.genekeyes.com/DW/DW-1.html
- Gene Keyes's map sources also made available in OOo draw format (need GIS knowledge) : http://www.genekeyes.com/MEGAMAP-BETA-2/
- CIA's factbook (less detailed but up to date) : https://www.cia.gov/library/publications/download/‎

Raster maps (not ready to use but easy to setup) :
- http://www.paulillsley.com/gia/index.html

Among above vector maps, wikipedia map probably the one that readily editable. For inkscape users, 15MB file is quite common/average but 15MB of just polyline is really heavy one! I have been using Tpx for some months now and it fulfill my need of semi-technical drawing. When I try to open World98+ map above, I thought it will suffocated but Tpx handle this file with just 60MB of RAM. Trying to do this with inkscape will freeze your computer winamp's playback -_-! But then how to edit this file efficiently?

Well in Tpx the drawing area is "barely" movable and zoomable, redraw takes several seconds (at any zoom) but no redraw for undo though. So maybe with more powerful CPU it may do. But there is easier way. Since Tpx spend little memory, you don't need to worry of running out of memory (how convenient).

Tpx don't have layering system like inkscape nor understand xref like CAD. So we do chunked file management.

- break the groups

 - copy-paste each region into new session of Tpx

remember that copy-pasted chunk will preserve coordinate/location, so to merge back simply use copy paste operation again.

- the ideal editing workspace is at country/states level:

Lesson learned:

Good application is the one that have good combination of stability and features not bloatness, and part of the stability is smart memory management. Tpx may lack much of SVG features such as gradient ramp/mesh but talk about mission critical apps: "before you can edit you need to open it!". Sorry if I sounds like making a dig to inkscape, but damn they sure know how to make bloat. I want my old sodipodi back!

 Additional note: In case you wonder how the world98+.svg created from http://commons.wikimedia.org/wiki/File:World98.svg. The metadata says:

"Generator: Adobe Illustrator 14.0.0, SVG Export Plug-In . SVG Version: 6.00 Build 43363)"

yeah a real world app huh!?

The Moluccas 0.8

Version 0.8 is here. I now called it Moluccas (not to be confused as some kind of mollusca).  It can be grabbed at https://sourceforge.net/projects/tumagcc/files/0.8/

The installer is M08.exe and you can pick up which 7z packages to install (categorized) but at least "base" is mandatory. Put all in the same folder and you're ready to install. There is brief overview of what's inside in each package at the link above. It still lack of documentation though only man included.

Enjoy my biggest OSS Collection with build environment!
All suggestions are welcome.

Main changelog:

- Fixes many typos and silly mistakes when tested in Windows 7
- All executable flagged with LAA
- Simplified startup: run conemu to start (old mode console, no longer supported though possible)
- Isolation: this version use ReactOS's cmd completely so it can be run independently of Windows' cmd.
- Installation speedup: Installer will use all cores (1 core/package)
- Explorer integration is disabled to provide zero host modification during startup (can be enabled by invoke molumenu). But many gui apps still write registry entries especially MFC based one.
- Fix integration with MSVS 2010 or (later?)
- Lots of manual tweaking in GUI apps to make it tight and blend in
- Small fixes in mingw headers and xtraposix libraries

Preparing version 0.8

I knew it will be more belated than previous version, this is getting bigger... Tuma MinGW 0.8 will focus on integration fixes and aim to become more established collection. On gcc side there is very little changes and will be the last with 4.7.x series (barebone 4.8.1 is provided though), originally I want to bundle static gtk libraries, but decided that's not worthy enough (I will put it up separately).

This version will have serious GUI apps addition and I have evaluate dozens of FOSS each weekend to find apps that lean and fast (avoiding .net/java whenever possible). But they're just alternative workflow from commandline counterpart which has considerable advatage. I still not quite done with cleanup so I dumped those in extratool package.

Oh yeah I want to drop the name, I think it's not a problem since it's not popular at all anyway. Candidates: celebes, moluccas or rangrang (the last is meaningful cause I love ants, they are the oldest architect)

I hope could finish it on sunday tomorrow.

Node.js with posix path support

It was Mat Sutcliffe who sent me a patch to compile node.js with mingw last weekend, if I recalled node.js used to be compilable with mingw via scons but apparently they switch to gyp entirely and mingw support became broken since then. I though it was a bit weird to sent it to me, but it turn out the process would involve msys python (needed by gyp to work properly).

The patch itself is a win-win solution as it will build native win32 binary but with support for posix path (MSYS and CYGWIN) via generic mount mechanism. He further explain why he choose mount over cygpath, that is because node.js operate asynchronously so by using mount path would settled once rather repeatedly as with cygpath.

This is binary of node.js 0.10.12 (built as ./configure --gdb --without-etw --without-perfctr)
nodejs-0.10.12.7z
original patch for mingw by Mat -> replace msysmnt with mount
modified patch for mingw-w64
python msys

The functionality is implemented in path.js which is bundled inside the executable. So actually you could modify (if you wish) that part (look for mingwToWindowsPath) with your favorite hex editor if you don't want to recompile. Just make sure you don't add and reduce the string block length. Or if you think mingw is such mundane task feel free to apply patch only to path.js and then compile it with M$VS as usual :-)

How to build with mingw:
1. extract the source file for node.js homepage
2. download python for msys and extract to msys installation
3. download one of appropriate patches (above) put in source tree
3. start mingw, navigate to source tree and apply patch
4. ./configure [your options] (make sure that python for msys is the active one)
5. python tools/gyp_node -f make
6. LINK=g++ make -C out BUILDTYPE=Release V=1

Notes:
- I'm not sure about etw and perfctr whether those compilable with mingw but both don't compile out of box. 
- The download is contain official installation hierarchy with node.exe replaced. Since it's not installer (it make no sense anyway) I'll assume you know what to do next ;-)

Enjoy!

Pinpoint, another approach for presentation

Previously I've tried fluxus for live presentation using a basically game engine + scheme scripting. Due to unresolved issue and the need to learn new language, I'm giving up and looking for much simpler alternative despite I still consider fluxus as a novel apps with no equal alternative.

Pinpoint is small, light and opengl accelerated presentation tool with a kind of magicpoint approach (you write down tagged text representation). It's similar to OSG's Present3D which is between pinpoint and fluxus in term of features, but Present3D require you to write a not-that-readable XML file instead which is nasty without the help of WYSIWYG editor. Pinpoint is simple, it has no advanced timeline/layer like in office software and geared for quick and dirty presentation. Pinpoint also feature extendable transition FX system (powered by clutter) by writing your own FX via loadable json file.

Here you go:

Static build win32, patched to make it more standalone :-)
download: pinpoint and sample+patch

notes:
- to test just drag and drop introduction.pp to pinpoint.exe
- no video support (I'm still figuring out how to build gst-ffmpeg statically) seems played but not rendered by cluttersink
- image support: png, jpg, gif
- basic FX are bundled in the executable, if you make new fx put the json file alongside the executable
- export: pdf only
- pinpoint.exe has console window, pinpointw.exe doesn't
- if you want to compile it yourself, please use gnome libraries released circa 2011 and avoid cairo 1.12.x

see their homepage to learn how to write a presentation file https://live.gnome.org/Pinpoint
Enjoy!

CMD restriction bypass

Alright since I have skipped nearly the whole month, I need to buzz something at least. The tittle isn't that real, I mean I'm bored with black hack stuff at my age anyway, so this one is too obvious, legal and easy too.

How to bypass CMD restriction? Well, by using another compatible cmd of course, like the open source ReactOS (wine?) cmd.exe replacement from the ISO file (version 0.3.14 works with XP and 7). However the problem soon arise when COMSPEC get tangled in the way, for example standard system() call will still use system-wide COMSPEC (not session one). So things like for /f ... ('command') do ... wont work since it will ask blocked system's cmd.exe to launch something a.k.a %windir%\system32\cmd.exe /c bla bla.

I stumble on this case when I use my previous 7z-sfx wrapper for bat files. So I decide to put ReactOS's cmd.exe inside the sfx as workaround.

That also apply to regedit restriction which ReactOS also has the replacement, but of course there is no "such restriction" to just access registry whatsoever.