Tested gtm-reconnect

Now gtm-reconnect works with pgxc_ctl c version.

Work inprogress

Today, did the following:

1. Resolution of 9.2.3 merge with XC master.
2. Test and debug pgxc_ctl C-version.  Now it can configure all the slaves.  Failover test will be done next.

Need to write the book chap1.

WAL Filtering for table rebalancing

Now XC team finished table rebalancing feature by ALTER TABLE ... DISTRIBUTE BY ... statement.   This runs with all the rows of the table locked.    The next is to do this concurrently, not locking rows and run everything background.   Different from other ALTER TABLE operation, this doesn't affect the logical view of the table and is safe to run this concurrently, as in the case of CREATE INDEX CONCURRENTLY.

To implement this, I thought to use a kind of log shipping for specific table.   To choose WAL records for the target table and then send them to the handler.   This looks very simple but we should consider TOAST as well.    Some says that it's much simpler to implement this as a part of walsender.  Yes, this can be if we don't have to worry about full page writes.

Full page writes are not simple to deal with.   We have to extract logical operation from them.   We can avoid this difficulty if the WAL filter is a part of XLogInsert(), which runs as each backend process and we need to handle this as well.

Which will be better?

XC conversation

XC developers ML received a comment on table rebalancing from a new guy.   His comment shows that he is really using XC.   He is from Shanghai Linux User Group. Considering the number of XC downloads, many Chinese people must be using or testing XC.  Very interesting.

Hint clause in XC?

I came to this idea.   This could be a wicked idea though.   In the future, we will see many statement which we cannot generate the best plan.   So, to do this, we may need to accept a hint which distribution to rely on, as

SELECT * FROM A,B,C WHERE A.A=B.B and B.B=C.C A.A in (SELECT Z from X WHERE Y=100) /*+ USE_PUSHDOWN TO ALL */

Looks tempting ....

XC at PGCon2012

Now PGCon2012 schedule is out.   Postgres-XC will have tutorial for building, installing, configuring and running Postgres-XC database cluster.   This is both for users and developers.    Developers will learn XC's source code configuration and documentation internals, although the tutorial will be best-tuned for users.
Tutorial will include everything they should know to use XC, including node configuration and table design.
Scheduled on May 16th, Wed., 1:00PM to 4:00PM.

Testing XC toward 1.0

Now we're becoming the goal of this quarter's development.   Depending upon each mender's schedule, we will go into dedicated test for 1.0.
So far, from the discussion among the core members, we agreed that the test should consider at least three points of view.

1. How many bugs are hidden and how many bugs should be found.    Considering the written code is around 100,000 lines, I think we should find around 500 bugs.    I do hope the code is much better.   We need much more test cases to run.
2. SQL functionality check: what is supported and what is not.    I've been discussion this with the sponsor.   I don't know if the developer team should (and can) do this.    From my experience, it will take at least five to six man-month to write a doc of test case for this purpose.
3. Code test coverage.   There could be a handy tool to measure this.   Although test code test coverage doesn't tell about the condition for each piece of code, Pavan tells this will be good to check the coverage of error handling.   I agree on this.

I should write first draft of XC test schedule by the next Tuesday to discuss in core member's teleconference.

After consideration...
Maybe increase code coverage makes most sense from current development team.   Feature test and performance test are being done for each development project.

Cluster Summit and PGCon

Now began discussion about Cluster Summit in PGCon with Josh.   Josh still likes an idea to have CHAR in Europe, Japan and US in turn.    Yes, I think this is quite a good idea but it seems to me that we need some independent organization (in PostgreSQL community) to manage this, as well as local organization to help preparation.     In Feb, we may be able to innate this to Simon at Paris through Michael.    Should Japanese local organization be in JPUG?

pgxc_clean

Finished initial pgxc_clean code.  It is not tested yet.

At first, I thought I need direct connection to GTM to clean up TXN status, but finally I found that I just need to tell the coordinator to commit or abort prepared transaction.  With this, the code is simple enough.   It is just libpq application.   If I wish, I could have written this as ECPG application or even JDBC application.

Anyway, because this completely depends upon XC internal catalog and proprietary function calls, this is local to XC and is not portable at all.

GTM HA for 0.9.7

Committed the patch.   Now GTM's transaction backup to the standby is accurate, even in heavy workload.   Remaining issues are:

1. tcp_keepalives feature works to monitor client connection at the server.   Need some more work to enable monitoring server connection at clients.
2. Need to add "status" feature.   I once considered to add this to gtm_ctl.   It may work but new command can be okay too.    Anyway, this feature should be implemented using normal communication with gtm/gtm_proxy, not signal.
3. "gtminit", associated with XC cluster bootstrap.

GTM standby test

Found that GTM synchronous backup does not work well.
What found:

1. Try to sync with GTM-Standby by gtm_sync_standby()
2. It does not return somehow.

Before these, called bkup_node_register_internal() to backup the command to GTM-Standby and it looks successful.

Maybe GTM-Standby does not handle backup messages correctly and GTM-Standby is waiting for some more information.

GTM standby

Now GTM standby is at the last stage.

1. Error handling in GTM-Proxy done: when GTM communication error occurs, proxy does some retry and if it fails then wait for reconnect operation.
2. GTM standby can now connect to GTM at any moment.
3. After reconnect and shutdown whole cluster, GTM-Standby can start as GTM.   Do not forget to rearrange configuration files properly.
4. GTM backup is now corrected so that transaction handle and GXID are backed up to the standby.
5. Asynchronous backup works fine.
6. Synchronous backup has still some issues.   Maybe some minor protocol handling.   Will be tested next.

XC bootstrap

Pavan wrote his proposal on XC bootstrap.   Here's some of my idea/comment

1. It's nice to run initdb as independently as possible and register each node after initdb.   It will be even nicer if initdb runs vacuum freeze so that any node can begin with any GXID.   It will make adding nodes safer.
2. He is right that XC configuration makes sense with at least one coordinator registered.   The issues are:
1. Should a coordinator registered at first?  The order of registration can be more flexible.   Because clients target to a coordinator, there will be no problem to have only datanodes initialized and registered at initial phase of the bootstrap.
2. Should datanode be registered to GTM?   What the registration works for?   
3. Coordinator was made a separate node because we thought coordinator and datanode should be different binaries.  Now that they share the same binary what happens if we a node is both coordinator and datanode?   If they're the same, configuration may look simpler.

GTM fourth improvement

Now I'm tackling the fourth GTM improvement, to correct backup algorithm.

Current implementation just proxies begin transaction/get GXID type command directly to the backup.   Backup tries to assign GXID and transaction handle independently and depending upon the order of each thread execution, each transaction can be assigned different gxid at GTM standby, which may cause serious problem when standby fails over.

The improvement corrents this.   Now both handle and gxid are backed up.   GTM standby will assign transaction struct slot based upon backed-up handle and use backed-up gxid to keep everything consistent.

One thought about sequence.   Because it is not practical to backup current sequence value, sequence command is essentially proxied to the backup (standby).   Because history of sequence value has very little meaning and requirement is to begin the next value correctly when failed over, I think current mechanism should work well.

More over, I eliminated needless response from Standby to ACT.    When synchronous backup is specified, acknowledgement will be exchanged to make sure that backups reached the standby.

Most of this is skipped when message comes through GTM-Proxy.  I need to find where I should insert the code to synchronize in this case.

----
(Addenda: 30th Dec., 2011)

1. Add "Backup_synchronously" to GTM_conn or GTM_TheadInfo

2. Check Backup_synchronously when flush to the client and "send_smething".   Send something to backup before send something to the client.

3. Check if I'm running in standby or not when accepting commands.  

Mac, Linux, Windows in the office

Bought new LCD display with 1080p resolution and now running Mac, Linux and Windows.   Mac book air runs pretty quick.   More than that, thanks to KeyRemap4, key-binding is set to emacs!   This is done in very low level and makes typing really a fun.

One tweak to use Mac (or other system) through HDMI.   HDMI picture is originally tuned for movies, or digital TV.   You should set this to PC ir still picture, which makes screen really beautiful.

My keyboards:

Happy Hacking Keyboard professional for Linux and Windows.   Really cool keyboard I've ever met.

Apple wireless keyboard for Mac book air clam shell mode.  I didn't expect much but found this keyboard is really as cool as HHK pro.

Happy typing!

GTM-standby third patch

Now third patch is done.   What I did today are:

1. Add missing option definition,
2. Correcting wrong option description
3. Adding log message of error detection, GTM connection retry and reconnect to GTM-standby.

Then I can test this code tomorrow!

GTM-Proxy fails with SEG-V

Sudo-san reported me that GTM-Proxy fails with SEG-V.    It runs normally in Ubuntu and without O2 build option, it also runs in CentOS too.   Finally, I found that it only fails with O2 option at CentOS.    It looks that entry to memory allocation handler is corrupted.   I will look into it tomorrow.

---
It was caused by uninitialized thrinfo, which points to all the memory context.   Just adding memset() fixed the problem.

I checked all the other malloc() in GTM-related code and found all the others are associated with proper initilization or written before read.

GTM-proxy error handling for reconnect

Added a code to allow GTM-Proxy to do the following:

1) Optionally retry connection to current GTM.   Specifies count, idle and interval.
2) Optionally waits reconnect command.   Specifies count, idle and interval.

To reduce the number of options, I'm not willing to introduce "option" flag to ask yes or no.   Instead, maybe we should consider that all zero menas no retry or no reconnect.   This code has not been done yet.  Maybe next Monday.

Anyway, need to log this activity.

----
Gee! After I made the first commit of this feature, I found there're couple of issues to be fixed before tested.

1) Description of new option GTM_OPTNAME_RETRY_IDLE ... This is wrong!!
2) GTM_OPTNAME_ERR_WAIT_IDLE definition is missing!!

I need to fix them as well as documentation.

Now, GTM_OPTNAME_ERR_WAIT_OPT is removed.   So, for connection retry and wait for reconnect,
if all the idle, count and interval are zero, then no such action will be made.

If communication error is detected and no retry, no reconnect wait are specified, then this will cause FATAL error because GTM-Proxy cannot continue service.

---
Koichi

GTM Standalone connectivity improvement (2)

Today, I finished the first code for this.   Tested the following:

1. start gtm
2. start gtm_proxies
3. start coordinators/datanodes and do some session through psql.
4. start gtm standby and do another psql sessions.

They worked okay.  Promote and reconnect seems to work fine.

However, when gtm standby is shutdown and try to connect again, gtm/gtm_proxies seems to stall.   Maybe disconnect sequence doesn't work well.  Need to fix this before commit.

GTM standalone connectivity improvement

Now GTM standby can connect and backup GTM when it connects before any other GTM client connects.   Here's an improvement.

1. GTM standby connects to GTM and send new command "BEGIN BACKUP" or something like this.   When GTM receives this command, GTM first lock all the other thread at their static points, typically just before they read request from clients.
2. GTM thread connected to GTM standby then establish connections to GTM-Standby, link this connection to thread information of each thread.  Then release locks so that other threads can continue to work with new backup connection to GTM standby.
3. Main thread does not have any connection to the standby.

Needed extension:

1. New entry in GTM thread information to indicate if the thread is main or not.
2. New command to indicate to "BEGIN BACKUP", and then "END BACKUP".  When "BEGIN BACKUP" is received, GTM will lock all the other threads.   When "END BACKUP" is received, new communication to GTM-Standby will be established for each worker thread (not for main thread), then all the thread locks will be released.
3. To lock each thread, thr_lock can be used.  This is defined and initialized but not used so far.