| Author |
|
BeachBum
Super User
Joined: April 11 2007
Location: United States
Online Status: Offline
Posts: 1880
|
| Posted: April 19 2008 at 13:44 | IP Logged
|
|
|
Lee I’m glad I’m paralleling your test with 2 systems just as backup. Let us know what you’ve done to recover. If you can’t don’t cycle PH come back here first so you won’t lose the data you obtained so far and maybe one is can get you around the problem.
__________________
Pete - X10 Oldie
|
| Back to Top |
|
| |
BeachBum
Super User
Joined: April 11 2007
Location: United States
Online Status: Offline
Posts: 1880
|
| Posted: April 20 2008 at 13:45 | IP Logged
|
|
|
Progress so far… PH and SDM memory usage still remains the same in that it goes up and down depending whether it remains minimize or restored then minimized. Ph Handles and Threads also basically remain the same. The overall system appears to be growing using Task Manager. It’s Memory, Handles and Threads continue to grow which leads back to an earlier thought I posted in that something else is making PH look bad. I’m using Process Explorer for program monitoring and using Task Manager for overall readings. Data is collected 3 times in a 24 hour period and has been running on 2 systems since April 16th from a fresh start. PH Insteon scan is turned off for the test. If the PLC will stay alive long enough I will continue to collect.
__________________
Pete - X10 Oldie
|
| Back to Top |
|
| |
grif091
Super User
Joined: March 26 2008
Location: United States
Online Status: Offline
Posts: 1357
|
| Posted: April 20 2008 at 15:00 | IP Logged
|
|
|
I reflashed the PLC coreApp using InHomeFre. Now have a running PLC. Searched through Smarthome forum to see if I could find a definitive conclusion regarding a hardware or firmware cause for PLC problem. Did not find a conclusion. A few had similar problems, resolved with the reflash of the coreApp, but no answer as to cause.
Pete, your no growth results when not scanning are consistent with a small 3 hour test I ran. I stopped my test once it was clear that if ph was not doing anything (I have no HA), ph and sdm3 were not growing.
These are the process memory numbers from 18 hours of testing before PLC stopped working...
smd3 ph
12350 25788 at the start of test
44692 26356 after 18 hours
Intermediate measurements suggest ph will/has leveled off around the +1M high water mark. SDM3 was growing at a steady rate. Did not try the open/minimize sdm3 window sequence suggested as a possible way to get the SDM3 process to drop memory.
Have not restarted my long test. Guess I will start that tonight with the same 10 second scan and hope the PLC does not stop working again.
__________________
Lee G
|
| Back to Top |
|
| |
Gadgets
Senior Member
Joined: January 28 2008
Location: Canada
Online Status: Offline
Posts: 178
|
| Posted: April 21 2008 at 00:41 | IP Logged
|
|
|
Hi guys
Been reading through all of this memory issue. I do not have any input for the technical side of this but had a thought if this is a real issue with either PH or SDM. Is there not a function in PH that can restart the entire system? I thought I came across that somewhere. If so PH could simply be configured to automatically reboot the system and re-load PH once a week if needed.
I only use old X10 stuff here right now none of these new fancy controllers so I don't have this problem. My system can run for weeks and weeks without issue (other then OE crashing).
Just a thought.... 
__________________
Friends, don't let friends install Norton Products
|
| Back to Top |
|
| |
BeachBum
Super User
Joined: April 11 2007
Location: United States
Online Status: Offline
Posts: 1880
|
| Posted: April 21 2008 at 08:58 | IP Logged
|
|
|
Tony did some work on restore and minimize SDM but with further testing it did not work as intended. So far in my testing the results do not support a memory leak with PH nor SDM. As far as bouncing the system I have to issue a HALTDM command to the PLC so I can shutdown smoothly. There are some RUN line commands that will work also. My experience is that the PLC will lock up before there is a foreseeable memory problem.
__________________
Pete - X10 Oldie
|
| Back to Top |
|
| |
grif091
Super User
Joined: March 26 2008
Location: United States
Online Status: Offline
Posts: 1357
|
| Posted: April 21 2008 at 13:51 | IP Logged
|
|
|
Here is the process memory data I collected before PLC hung up between the 18 and 19 hour mark. Just about where it stopped working during the last test except this time it did not blink at the twice per second rate and it came back after unplug/replug. Restarting ph and sdm3 did not restore PLC to operation. ph restarted sdm3 a few times when the isResponding failed after I restarted ph.
SDM3 ph rteng7
4/20 6;10 pm 12480 25828 7852
9:05 pm 19660 26236 14252
10:02 pm 22040 26244 14256
4/21 2:30 am 32708 26260 14268
6:15 am 41824 26268 14268
8:37 am 47168 26268 14268
10:45 am 52272 26288 14268
12:45 am 56784 26360 14268
1:15 pm 57036 26364 14268 all sends failed
1800 after open/min smd3 window
As with my previous test, ph was scanning at a poll rate of 10 seconds. Looks like ph memory stabilized, sdm3 continued to climb. With Pete's results previously posted, I would say the activity associated with scanning produced the sdm3 increase. I'm finished with my measurements.
__________________
Lee G
|
| Back to Top |
|
| |
BeachBum
Super User
Joined: April 11 2007
Location: United States
Online Status: Offline
Posts: 1880
|
| Posted: April 21 2008 at 13:55 | IP Logged
|
|
|
Lee, I not showing SDM growing it is holding around 12 -14. I’m using 235V
Interesting note I am also tracking Private Virtual Memory. By definition from Process Explorer: “Private Bytes represents the amount of private virtual memory a process has allocated and is the value that will rise of a process exhibiting a memory leak bug.” Private Virtual Memory has grown from 29.3 MB to 39.1 in 5 days. Working Set Memory seems to average out somewhere around 18 MB. Am I interpreting this correctly Brian?
__________________
Pete - X10 Oldie
|
| Back to Top |
|
| |
grif091
Super User
Joined: March 26 2008
Location: United States
Online Status: Offline
Posts: 1357
|
| Posted: April 21 2008 at 14:01 | IP Logged
|
|
|
Pete, did you not say earlier that you had turned scanning off. If I turn off scanning, I do not see sdm3 grow.
EDIT: I am using 308 as the earlier version does not work with Houselinc
Edited by grif091 - April 21 2008 at 14:02
__________________
Lee G
|
| Back to Top |
|
| |
BeachBum
Super User
Joined: April 11 2007
Location: United States
Online Status: Offline
Posts: 1880
|
| Posted: April 21 2008 at 14:03 | IP Logged
|
|
|
Scan is off...
__________________
Pete - X10 Oldie
|
| Back to Top |
|
| |
grif091
Super User
Joined: March 26 2008
Location: United States
Online Status: Offline
Posts: 1357
|
| Posted: April 21 2008 at 14:12 | IP Logged
|
|
|
I thought I had read that in one of your previous posts. With scan off, how much Insteon activity does ph produce in an average hour? When you run ph to have your HA active, do you run with scan on or off.
__________________
Lee G
|
| Back to Top |
|
| |
BeachBum
Super User
Joined: April 11 2007
Location: United States
Online Status: Offline
Posts: 1880
|
| Posted: April 21 2008 at 14:27 | IP Logged
|
|
|
I basically don’t do linking except when I install new devices. My groups are primarily for the purpose of backup if I lose the system. If I’m not testing I run it only if no lights are on and its daylight and that’s once per minute. As far as Insteon traffic any device change will trigger a run to all the devices to check their levels. So when nothing is happening it gets quiet but 30 minutes before Sunset or a dark day things begin to trigger rapidly. The only things I do manually are turn on/off the TV and go to the beach.
__________________
Pete - X10 Oldie
|
| Back to Top |
|
| |
bhlonewolf
Senior Member
Joined: March 29 2007
Online Status: Offline
Posts: 198
|
| Posted: April 23 2008 at 19:26 | IP Logged
|
|
|
I'm using SDM 308 and the size isn't growing so seeing that trend Lee posted above is alarming. But because it's not across the board, seems like it's a compat or driver issue somewhere. The trouble is, even if it continues, about the best you could hope for is to find a work around (kind of like how PH handles the SDM now). I realize this is kind of a hack, because my instinct is to rebuild the dang machine and debug it to death before I'd just concede, but in this case if the leak isn't severe enough I'd schedule a "kill sdm3.exe" job to run once a week or so, and let PH restart it. I know, ugly, huh?
So BeachBum, as to your points -- yep, you're right. Watching the private virtual will conclusively tell you if there's a leak or at least a design problem, since that memory is allocated only to that process. It's possible to leak shared memory, too, but impossible (well, realistically impossible) to track down unless you're motivated.
|
| Back to Top |
|
| |
grif091
Super User
Joined: March 26 2008
Location: United States
Online Status: Offline
Posts: 1357
|
| Posted: April 23 2008 at 19:32 | IP Logged
|
|
|
How many Insteon devices do you have installed, are you scanning and what is the poll interval.
__________________
Lee G
|
| Back to Top |
|
| |
BeachBum
Super User
Joined: April 11 2007
Location: United States
Online Status: Offline
Posts: 1880
|
| Posted: April 23 2008 at 19:51 | IP Logged
|
|
|
Brian, what alarms me is the Virtual Private has gone from 15.4 MB to 42.8MB is 7 days while the working set stays at 18MB average. Should Dave be concerned??
__________________
Pete - X10 Oldie
|
| Back to Top |
|
| |
bhlonewolf
Senior Member
Joined: March 29 2007
Online Status: Offline
Posts: 198
|
| Posted: April 23 2008 at 22:38 | IP Logged
|
|
|
OK, I admit I wasn't following my own advice and just relying on Task Manager (working set) view of the data. To be honest, I run a huge SQL Server DB on the machine so the mem usage by anything else is simply dwarfed by comparison. So I've been looking at the private usage exclusively and it is troubling, but too early to draw conclusions... will update.
Pete -- yes and no on that. You're referring to the SDM? If that's the case, unfortunately out of Dave's hands entirely since it's Smarthome. But back the issue:
Because the working set is flat and virtual private grows, to me that seems to suggest the majority of the data the app has reserved isn't getting referenced, otherwise the working set would be higher. I don't want to conclude it's a leak, but it's certainly the right trend.
So here's one way to look at it: you've got a growth of ~5MB/day for the SDM. Over a month, it's obviously 150MB, over 4 months, it's 600MB. Certainly a lot but most of that will get paged anyway, and besides, I'd reboot the server long before 4 months anyway.
Or, you could simply have PH kill and restart the SDM process -- that should fix 'er up.
So I admit I'm pretty obsessive at finding and investigating these issues, but you know, if the system works otherwise, I can live with it :)
|
| Back to Top |
|
| |
BeachBum
Super User
Joined: April 11 2007
Location: United States
Online Status: Offline
Posts: 1880
|
| Posted: April 23 2008 at 23:05 | IP Logged
|
|
|
But the SDM is stable and it is PH I’m talking about. PH Private Virtual just keeps growing. After I get enough stats I’ll try some isolation techniques to get to the culprit.
__________________
Pete - X10 Oldie
|
| Back to Top |
|
| |
bhlonewolf
Senior Member
Joined: March 29 2007
Online Status: Offline
Posts: 198
|
| Posted: April 23 2008 at 23:38 | IP Logged
|
|
|
Oooh ok, I thought you were talking SDM. Sorry, different threads here going on.
Yeah, if it's PH, definitely try to isolate -- for example, if you disable the controllers and thus no insteon, does it grow? Etc.
Runtimes like powerbuilder (such as Java, .NET, etc.) are typically pretty resilient to memory leaks, but it's possible. Is this on the alpha by any chance? I believe dave is using a newer pb runtime with it -- might make a difference.
|
| Back to Top |
|
| |
grif091
Super User
Joined: March 26 2008
Location: United States
Online Status: Offline
Posts: 1357
|
| Posted: April 23 2008 at 23:42 | IP Logged
|
|
|
Brian, I run with scan on, using a 10 second polling value. With a minimum of 6 Insteon messages per minute, 360 Insteon messages per hour every hour,......, I see an sdm3 result (memory growth) that Pete does not see because he runs with scan off. That is why I asked if you run with scan on or off. When I turn scan off, I don't see the sdm3 memory growth because I don't have enough native Insteon traffic to show a problem. I can't keep my PLC running long enough to track the ph process growth. There is some growth during the 18 hours but not enough time to get results compared with Pete's runs. With a stable working set but an increasing total memory allocation to the ph process, it does sound like there is some memory management issue. May not be under programmer control with a 4GL language but it does seem like something is not right. It will be interesting to see if the next Beta Dave releases will have any change in memory management as he is moving to a later level of PowerBuilder (I think).
EDIT: I am running on the latest Beta.
Edited by grif091 - April 23 2008 at 23:44
__________________
Lee G
|
| Back to Top |
|
| |
bhlonewolf
Senior Member
Joined: March 29 2007
Online Status: Offline
Posts: 198
|
| Posted: April 24 2008 at 00:42 | IP Logged
|
|
|
Agreed, Lee, sounds like something is off. You know, it's troubling that you can't keep the PLC running that long -- potential memory leak aside.
I run with scan on, 30 second interval. A good number of triggers/etc. I also scan links, but don't do level/x10 scans.
So in the meantime, treating stability as a separate issue -- any ideas why you can't keep it running longer? Does it just seem to lock up?
|
| Back to Top |
|
| |
grif091
Super User
Joined: March 26 2008
Location: United States
Online Status: Offline
Posts: 1357
|
| Posted: April 24 2008 at 04:33 | IP Logged
|
|
|
My two ph tests ran for between 18 & 19 hours. The first test I started to get a Send Error (I think those were the words) in the sdm log window, with the PLC status LED blinking approx 2 times a second. Tried unplugging the PLC for a minute and replugging, then tried holding the Set button while I replugged the PLC. Tried holding the Set button after power was restored, tapping the Set button... Every time power was restored, the PLC status LED would return to blinking twice per second. Searched the Smarthome forum and found a post with that exact symptom. The proposed solution, which worked for the original poster, and myself, was to reflash the coreApp with some free software from InHomeFre, using an image from the Smarthome web site.
The second test a few days later ran for between 18 and 19 hours, with a Send Failed message (I think) in the sdm log window. This time the PLC status LED was not blinking twice per second. Restarted ph several times. Each time ph encountered an error communicating with the PLC and restarted sdm3. This time I just unplugged the PLC for 60 seconds, restored power and things were fine. No need to reflash the PLC this time.
With the symptom (albeit slightly different) happening at about the same point in both tests, I am thinking it is related to the number of Insteon messages processed during the test. I have no HA so, except for normal house light usage during the test, the Insteon message count is going to be 6 per minute x 60 minutes x 18+ hours.
The PLC is about two years old. It came in a combo pack when I purchased Houselinc Desktop from Smarthome. Never had a PLC problem in those two years. I plugged it in, connected it to a USB port and it has run fine from day one. I thought about buying a new PLC to see if that would correct the situation but Smarthome is out of stock with a restock estimate of 5/30/2008.
I have considered the possibility that there is some Insteon message limit associated with my PLC. I live in a rural area, where a power drop a few times a year is not unusual. Could be those power outages have reset the PLC so I never encountered a maximum message limit until running ph with the scan option.
My plan is to wait for Dave to move the PLM support into a Beta, pick up a PLM and try that. I believe the PLM is the correct long term strategy, regardless of my PLC hangups. ph is running fine for managing my Insteon network as far as setting links and device options go. Except for my curiosity, there is no pressing need to find an answer to the PLC question or possible memory leak. The new Beta will move me away from the PLC/sdm3 and I think will be on a later PowerBuilder release.
At that point I will reevaluate ph running 24/7. I have lots to learn about the HA function of ph so the time to get to a new Beta with PLM support will be used to "play" with HA.
I appreciate everyone's interest in this. Any ideas of things to try before the new Beta and PLM arrive, I'd be happy to try.
__________________
Lee G
|
| Back to Top |
|
| |
Active Topics 
Memberlist 
Search 
Help
Register 
Login
PowerHome Messageboard : PowerHome General
Topic: Memory Leak and Handle Count
