What is the Impact on the CPU Statistic in a 10046 Trace File in a CPU Constrained Server?

17 03 2010

March 17, 2010

Sometimes I have wondered about the statistics that appear in 10046 trace files – how much are the statistics distorted by other activities happening in the server, or even just the act of enabling a 10046 trace for a session.  Will the CPU statistics in the trace file increase as running processes remain in the CPU run queue longer and longer, or are the CPU statistics immune to longer execution times caused by a process’ “run” time slice expiring?  Does the CPU architecture matter?  Does the operating system matter?  Is it possible that a query could actually complete faster when the competition for CPU resources increases?  Are there other variables that need to be considered?

We will set up the same test table that was used in yesterday’s blog article:

CREATE TABLE T5 (
  C1 VARCHAR2(10),
  C2 VARCHAR2(100),
  PRIMARY KEY (C1));

INSERT INTO T5 NOLOGGING
SELECT
  'A'||TO_CHAR(ROWNUM,'0000000'),
  RPAD('A',100,'A')
FROM
  DUAL
CONNECT BY
  LEVEL<=1000000;

COMMIT;

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T5',CASCADE=>TRUE)

From a client computer the following script is executed against Oracle 11.1.0.6 running on 64 bit Linux:  

SET AUTOTRACE TRACEONLY STATISTICS
SET ARRAYSIZE 100
SET TIMING ON
ALTER SESSION SET TRACEFILE_IDENTIFIER = 'NOLOAD';
ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';

SELECT /*+ USE_NL(T51 T52 T53 T54 T55 T56 T57 T58) */
  T51.C1,
  T51.C2,
  T52.C1,
  T52.C2,
  T53.C1,
  T53.C2,
  T54.C1,
  T54.C2,
  T55.C1,
  T55.C2,
  T56.C1,
  T56.C2,
  T57.C1,
  T57.C2,
  T58.C1,
  T58.C2
FROM
  T5 T51,
  T5 T52,
  T5 T53,
  T5 T54,
  T5 T55,
  T5 T56,
  T5 T57,
  T5 T58
WHERE
  T51.C1=T52.C1
  AND T51.C1=T53.C1
  AND T51.C1=T54.C1
  AND T51.C1=T55.C1
  AND T51.C1=T56.C1
  AND T51.C1=T57.C1
  AND T51.C1=T58.C1
  AND T51.C1 BETWEEN 'A 0000000' AND 'A 1000000';

SELECT SYSDATE FROM DUAL;

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT OFF';

The SQL*Plus session displayed the following statistics, which were fairly consistent across multiple executions:

1000000 rows selected.

Elapsed: 00:01:59.85

Statistics
---------------------------------------------------
          0  recursive calls
          0  db block gets
    8696435  consistent gets
          0  physical reads
          0  redo size
   86861789  bytes sent via SQL*Net to client
     110349  bytes received via SQL*Net from client
      10001  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
    1000000  rows processed

TKPROF output for the trace file:

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        2      0.00       0.00          0          0          0           0
Execute      2      0.00       0.00          0          0          0           0
Fetch    10001     35.53      35.83          0    8696435          0     1000000
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    10005     35.53      35.83          0    8696435          0     1000000

Misses in library cache during parse: 0
Parsing user id: 80 

Rows     Row Source Operation
-------  ---------------------------------------------------
1000000  NESTED LOOPS  (cr=8696435 pr=0 pw=0 time=294720 us)
1000000   NESTED LOOPS  (cr=7696435 pr=0 pw=0 time=272839 us cost=8008211 size=888000000 card=1000000)
1000000    NESTED LOOPS  (cr=7457834 pr=0 pw=0 time=250538 us cost=7007667 size=777000000 card=1000000)
1000000     NESTED LOOPS  (cr=6219233 pr=0 pw=0 time=210922 us cost=6007124 size=666000000 card=1000000)
1000000      NESTED LOOPS  (cr=4980632 pr=0 pw=0 time=170866 us cost=5006580 size=555000000 card=1000000)
1000000       NESTED LOOPS  (cr=3742031 pr=0 pw=0 time=130944 us cost=4006037 size=444000000 card=1000000)
1000000        NESTED LOOPS  (cr=2503430 pr=0 pw=0 time=90395 us cost=3005494 size=333000000 card=1000000)
1000000         NESTED LOOPS  (cr=1264829 pr=0 pw=0 time=49809 us cost=2004950 size=222000000 card=1000000)
1000000          TABLE ACCESS FULL T5 (cr=26228 pr=0 pw=0 time=4775 us cost=4407 size=111000000 card=1000000)
1000000          TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=2 size=111 card=1)
1000000           INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000         TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000          INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000        TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000         INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000       TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000        INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000      TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000       INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000     TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000      INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000    INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000   TABLE ACCESS BY INDEX ROWID T5 (cr=1000000 pr=0 pw=0 time=0 us cost=1 size=111 card=1)

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                   10001        0.00          0.01
  SQL*Net message from client                 10001        0.01         83.34
  SQL*Net more data to client                 10000        0.00          0.19

The above shows that the 1,000,000 rows from the query were returned to the client in just less that 2 minutes.  8,696,435 consistent gets were performed with 35.53 CPU seconds, 35.83 elapsed seconds, and 83.34 seconds waiting for the next fetch request from the client computer.

For fun, we will start up 40 instances of the CPU load generating bash script found on page 197 of the Expert Oracle Practices book.  That will easily push the 8 processors fairly close to 100% utilization with fairly long CPU run queues.  Repeating the previous test, we receive the following output:

1000000 rows selected.

Elapsed: 00:01:57.01

Statistics
---------------------------------------------------
          0  recursive calls
          0  db block gets
    8696435  consistent gets
          0  physical reads
          0  redo size
   86861789  bytes sent via SQL*Net to client
     110349  bytes received via SQL*Net from client
      10001  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
    1000000  rows processed

TKPROF Output:

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        2      0.00       0.00          0          0          0           0
Execute      2      0.00       0.00          0          0          0           0
Fetch    10001     32.64      32.64          0    8696435          0     1000000
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    10005     32.64      32.64          0    8696435          0     1000000

Misses in library cache during parse: 0
Parsing user id: 80 

Rows     Row Source Operation
-------  ---------------------------------------------------
1000000  NESTED LOOPS  (cr=8696435 pr=0 pw=0 time=258421 us)
1000000   NESTED LOOPS  (cr=7696435 pr=0 pw=0 time=239249 us cost=8008211 size=888000000 card=1000000)
1000000    NESTED LOOPS  (cr=7457834 pr=0 pw=0 time=219902 us cost=7007667 size=777000000 card=1000000)
1000000     NESTED LOOPS  (cr=6219233 pr=0 pw=0 time=185868 us cost=6007124 size=666000000 card=1000000)
1000000      NESTED LOOPS  (cr=4980632 pr=0 pw=0 time=151577 us cost=5006580 size=555000000 card=1000000)
1000000       NESTED LOOPS  (cr=3742031 pr=0 pw=0 time=117680 us cost=4006037 size=444000000 card=1000000)
1000000        NESTED LOOPS  (cr=2503430 pr=0 pw=0 time=83916 us cost=3005494 size=333000000 card=1000000)
1000000         NESTED LOOPS  (cr=1264829 pr=0 pw=0 time=49146 us cost=2004950 size=222000000 card=1000000)
1000000          TABLE ACCESS FULL T5 (cr=26228 pr=0 pw=0 time=5678 us cost=4407 size=111000000 card=1000000)
1000000          TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=2 size=111 card=1)
1000000           INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000         TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000          INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000        TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000         INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000       TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000        INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000      TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000       INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000     TABLE ACCESS BY INDEX ROWID T5 (cr=1238601 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000      INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000    INDEX UNIQUE SCAN SYS_C0015042 (cr=238601 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 74553)
1000000   TABLE ACCESS BY INDEX ROWID T5 (cr=1000000 pr=0 pw=0 time=0 us cost=1 size=111 card=1)

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                   10001        0.00          0.01
  SQL*Net message from client                 10001        0.01         83.33
  SQL*Net more data to client                 10000        0.00          0.27

From the above we see that the 1,000,000 rows were returned to the client almost 3 seconds faster than before with essentially no change in the SQL*Net message from client wait – the statistics show that the CPU time was reduced by almost 3 seconds.  Great – the next time my SQL statement is burning CPU, I’ll just run 40 other CPU consuming processes to make my SQL statement run faster.  :-)  (Nehalem magic?)

I am sure a couple of people are fans of running Oracle on Windows, so let’s try the test on Oracle 11.1.0.7 running on 64 bit Windows with no other CPU load:

1000000 rows selected.

Elapsed: 00:01:38.40

Statistics
---------------------------------------------------
          1  recursive calls
          0  db block gets
    8695518  consistent gets
          0  physical reads
          0  redo size
   86861789  bytes sent via SQL*Net to client
     110349  bytes received via SQL*Net from client
      10001  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
    1000000  rows processed

TKPROF Output:

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.07       0.09          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch    10001     12.23      12.93          0    8695518          0     1000000
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    10003     12.30      13.02          0    8695518          0     1000000

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 56 

Rows     Row Source Operation
-------  ---------------------------------------------------
1000000  NESTED LOOPS  (cr=8695518 pr=0 pw=0 time=187370 us)
1000000   NESTED LOOPS  (cr=7695518 pr=0 pw=0 time=156136 us cost=8008160 size=888000000 card=1000000)
1000000    NESTED LOOPS  (cr=7457048 pr=0 pw=0 time=156136 us cost=7007623 size=777000000 card=1000000)
1000000     NESTED LOOPS  (cr=6218578 pr=0 pw=0 time=156136 us cost=6007087 size=666000000 card=1000000)
1000000      NESTED LOOPS  (cr=4980108 pr=0 pw=0 time=31242 us cost=5006551 size=555000000 card=1000000)
1000000       NESTED LOOPS  (cr=3741638 pr=0 pw=0 time=31242 us cost=4006015 size=444000000 card=1000000)
1000000        NESTED LOOPS  (cr=2503168 pr=0 pw=0 time=0 us cost=3005479 size=333000000 card=1000000)
1000000         NESTED LOOPS  (cr=1264698 pr=0 pw=0 time=0 us cost=2004943 size=222000000 card=1000000)
1000000          TABLE ACCESS FULL T5 (cr=26228 pr=0 pw=0 time=0 us cost=4407 size=111000000 card=1000000)
1000000          TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=2 size=111 card=1)
1000000           INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000         TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000          INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000        TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000         INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000       TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000        INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000      TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000       INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000     TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000      INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000    INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000   TABLE ACCESS BY INDEX ROWID T5 (cr=1000000 pr=0 pw=0 time=0 us cost=1 size=111 card=1)

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                   10001        0.00          0.00
  SQL*Net message from client                 10001        0.01         84.86
  SQL*Net more data to client                 10000        0.00          0.42

Nice, completed 21.5 seconds faster than the no load test on Linux/Oracle 11.1.0.6 running on the same hardware, even with a hard parse.  The CPU utilization is also 1/3 of the Linux test results. 

Now let’s load up the server with 40 CPU consuming processes using the VBS script found on page 197 of the Expert Oracle Practices book.  The test results follow:

1000000 rows selected.

Elapsed: 00:01:47.39

Statistics
---------------------------------------------------
          0  recursive calls
          0  db block gets
    8695518  consistent gets
          0  physical reads
          0  redo size
   86861789  bytes sent via SQL*Net to client
     110349  bytes received via SQL*Net from client
      10001  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
    1000000  rows processed

TKPROF Output:

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch    10001     22.24      22.49          0    8695518          0     1000000
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    10003     22.24      22.49          0    8695518          0     1000000

Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 56 

Rows     Row Source Operation
-------  ---------------------------------------------------
1000000  NESTED LOOPS  (cr=8695518 pr=0 pw=0 time=280798 us)
1000000   NESTED LOOPS  (cr=7695518 pr=0 pw=0 time=249598 us cost=8008160 size=888000000 card=1000000)
1000000    NESTED LOOPS  (cr=7457048 pr=0 pw=0 time=218398 us cost=7007623 size=777000000 card=1000000)
1000000     NESTED LOOPS  (cr=6218578 pr=0 pw=0 time=218398 us cost=6007087 size=666000000 card=1000000)
1000000      NESTED LOOPS  (cr=4980108 pr=0 pw=0 time=187198 us cost=5006551 size=555000000 card=1000000)
1000000       NESTED LOOPS  (cr=3741638 pr=0 pw=0 time=155998 us cost=4006015 size=444000000 card=1000000)
1000000        NESTED LOOPS  (cr=2503168 pr=0 pw=0 time=124799 us cost=3005479 size=333000000 card=1000000)
1000000         NESTED LOOPS  (cr=1264698 pr=0 pw=0 time=31200 us cost=2004943 size=222000000 card=1000000)
1000000          TABLE ACCESS FULL T5 (cr=26228 pr=0 pw=0 time=0 us cost=4407 size=111000000 card=1000000)
1000000          TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=2 size=111 card=1)
1000000           INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000         TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000          INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000        TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000         INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000       TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000        INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000      TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000       INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000     TABLE ACCESS BY INDEX ROWID T5 (cr=1238470 pr=0 pw=0 time=0 us cost=1 size=111 card=1)
1000000      INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000    INDEX UNIQUE SCAN SYS_C0016378 (cr=238470 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 70312)
1000000   TABLE ACCESS BY INDEX ROWID T5 (cr=1000000 pr=0 pw=0 time=0 us cost=1 size=111 card=1)

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                   10001        0.00          0.00
  SQL*Net message from client                 10001        0.01         83.55
  SQL*Net more data to client                 10000        0.00          0.59

The Windows test required an extra 9 seconds of CPU time when 40 processes were competing for the CPUs.  The CPU time for the fetch calls increased by 10 seconds over the test run with no competing processes.  Interestingly, the SQL*Net message from client wait decreased by a second during the test run with the increased CPU usage; I suppose that this might be expected considering that the next request from the client might have arrived while the Oracle thread was waiting in the CPU run queue.  Even with the additional 10 seconds of CPU time, the CPU  time consumed was still 10.5 seconds lower than that experienced with the Linux and Oracle 11.1.0.6 combination.  So, is the drop in CPU utilization an improvement in Oracle 11.1.0.7 that yielded better CPU times (you probably saw what happened when the same SQL statement was executed during yesterday’s blog article), is 64 bit Windows just that much more efficient than 64 bit Linux, or do you think that it was the 917 consistent get decrease that allowed Windows to complete the SQL statement faster than Linux?

The above test seems to indicate that Oracle on Linux does not include the time the process spent waiting in the CPU run queue when there is a lot of competition for CPU time – the fact that the rows returned to the client faster when the server was under a load probably means that more testing is required.  The above also seems to indicate that Oracle on Windows does inflate the CPU time statistic when competition for CPU time increases, but of course more testing should probably be performed.  So, why the recent increased interest in the CPU statistics in a trace file?  This OTN thread perked my interest.


Actions

Information

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Follow

Get every new post delivered to your Inbox.

Join 147 other followers

%d bloggers like this: