A Project Fails Repeatedly

Issue: One or more projects in a cluster repeatedly shut down unexpectedly.

Solution: Consider increasing the heartbeat timeout value the cluster uses to ensure that projects are running. Note, however, that issues related to the heartbeat timeout setting are unusual; it should rarely be necessary to change it. The default project heartbeat timeout is 20000 milliseconds (20 seconds).

To confirm that the heartbeat timeout is the issue, check the project log in STREAMING_HOME/cluster/projects/<cluster-name>/<workspace-name>.<project-name>.<instance-number>/esp_server.log, or in a Studio cluster <your-home-dir>/SybaseESP/5.1/workspace/<workspace-name>.<project-name>.<instance-number>/esp_server.log
Look for a 722014 message that includes a last contact delta value. The message looks similar to this:
```
2013-02-22 01:20:55.036 | 12611 | container | [SP-2-722014] (5741.829)
      sp(12589) Manager.heartbeatApplication() asked to stop. Last contact delta=20568   
```
The delta value is the time in milliseconds between the final contacts between the project and the cluster. If the delta value is close to or larger than the heartbeat timeout value, try increasing the heartbeat timeout value.
In an editor, open the node configuration file, STREAMING_HOME/cluster/nodes/<node-name>/<node-name>.xml.

Replace the Manager element with this code:

<Manager enabled="true">
  <!-- The ApplicationHeartbeatTimeout node is optional -->
  <!-- The first Manager in the cluster determines the value cluster-wide -->
  <!-- The value is in milliseconds -->
  <ApplicationHeartbeatTimeout>20000</ApplicationHeartbeatTimeout>
</Manager>

Note This is the top-level Manager element, not a Manager element in the Cache | Managers section.

Replace the default value of ApplicationHeartbeatTimeout, 20000, with a value larger than the last contact delta found in your log. For example, to increase the timeout to 30 seconds, enter 30000.
Copy the new Manager section into the <node-name>.xml file for every manager node in the cluster.
Stop and restart the cluster, shutting down controller-only nodes first, then manager nodes, and starting all the manager nodes before the controller-only nodes.