| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-28 | |||
| 19:28:52 | smcginnis | Anyone aware of any known issues that would be causing this - http://logs.openstack.org/93/596493/1/check/cinder-tempest-dsvm-lvm-lio-barbican/d716acf/logs/screen-n-cpu.txt.gz#_Aug_27_14_31_35_678991 | |
| 19:30:48 | efried | mriedem, dansmith: Didn't y'all see this the other day ^ ? | |
| 19:31:01 | mriedem | no it was an instance.save() when i saw it | |
| 19:32:18 | smcginnis | Odd that it seems to be somewhere that it kills the whole service. | |
| 19:32:25 | mriedem | in this case, compute is asking the cell conductor for the min nova-compute service version in that cell, | |
| 19:32:30 | mriedem | and timing out waiting for a reply | |
| 19:32:40 | mriedem | smcginnis: it's on startup of the service - init the compute rpc api | |
| 19:32:59 | mriedem | i wonder if nova-compute is starting before nova-cond_cell1? | |
| 19:33:09 | mriedem | n-cond-cell1 i mean | |
| 19:33:22 | smcginnis | Was just checking timestamps there. | |
| 19:33:23 | openstackgerrit | Eric Fried proposed openstack/nova master: Do test_reshape with an actual startup https://review.openstack.org/597218 | |
| 19:33:47 | mriedem | n-cond-cell1 appears to be up around at least Aug 27 14:30:40.185973 | |
| 19:34:03 | mriedem | n-cpu fails at Aug 27 14:31:35.678991 | |
| 19:34:16 | smcginnis | Yeah, about a second ahead of the failure on another one I'm looking at too. | |
| 19:34:34 | mriedem | so n-cpu and n-cond-cell1 looks like they are starting at roughly the same time, | |
| 19:34:43 | mriedem | n-cpu starts querying n-cond-cell1 before it's fully up maybe | |
| 19:34:46 | mriedem | then times out | |
| 19:35:27 | smcginnis | Does that need to happen in __init__? Seems like maybe somewhere after where it could retry once or twice might be good. | |
| 19:36:24 | mriedem | doesn't probably have to no | |
| 19:36:29 | mriedem | we could do it in post_start_hook or something | |
| 19:36:32 | mriedem | not sure if it would help | |
| 19:37:15 | mriedem | self.compute_rpcapi is pretty much only used for inter-compute comms anyway | |
| 19:37:20 | mriedem | it's not used on startup | |
| 19:38:32 | mriedem | 2018-08-27 14:30:31.159 | + functions-common:_run_under_systemd:1482 : sudo systemctl start devstack@n-cond-cell1.service | |
| 19:38:52 | mriedem | 2018-08-27 14:30:32.414 | + functions-common:_run_under_systemd:1482 : sudo systemctl start devstack@n-cpu.service | |
| 19:39:19 | mriedem | i think using upgrade_levels=auto is relatively new in devstack, and i think i added it... | |
| 19:39:36 | smcginnis | I'm fine with blaming you. | |
| 19:40:01 | mriedem | https://github.com/openstack-dev/devstack/commit/21221d1ad1462cdcaed4d052c3324ae384b407d4 | |
| 19:40:17 | mriedem | been there since may so not new | |
| 19:40:32 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22_determine_version_cap%5C%22%20AND%20message%3A%5C%22MessagingTimeout%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 19:40:40 | mriedem | but we might not have multi-line indexing properly in this job | |
| 19:41:11 | mriedem | here we go http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22_determine_version_cap%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 19:41:26 | mriedem | smcginnis: can you report a bug for this so we can at least start tracking it in e-r? | |
| 19:41:40 | melwitt | internally, people have asked for retry behavior there, if nova-compute starts up before nova-conductor is available | |
| 19:41:54 | melwitt | I looked at it a bit but didn't see where/how we could add it | |
| 19:42:14 | smcginnis | OK, I'll file a bug so at least it's tracked. | |
| 19:42:31 | mriedem | i'd probably move it to post_start_hook, | |
| 19:42:35 | mriedem | add a 3-time retry on it, | |
| 19:42:41 | mriedem | fail if we still can't get the thing | |
| 19:42:52 | mriedem | threading it out doesn't help since then we can't kill the service | |
| 19:43:17 | mriedem | although i think you can just SIGHUP and get nova-compute back on track after n-cond-cell1 is runnign | |
| 19:43:31 | mriedem | i know a guy in portland that knows how all of this shit works | |
| 19:43:48 | melwitt | :) | |
| 19:48:24 | smcginnis | Lovely - launchpad timeout submitting bug. | |
| 19:49:05 | openstackgerrit | Eric Fried proposed openstack/nova master: reshaper gabbit: Nix comments re doubled max_unit https://review.openstack.org/597220 | |
| 19:49:53 | smcginnis | Done - https://bugs.launchpad.net/nova/+bug/1789484 | |
| 19:49:53 | openstack | Launchpad bug 1789484 in OpenStack Compute (nova) "n-cpu fails init on timeout calling n-cond-cell1" [Undecided,New] | |
| 19:55:53 | smcginnis | mriedem: This looks like it started on the 18th - http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22_determine_version_cap%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 20:06:14 | mriedem | efried: comments and nits inline https://review.openstack.org/#/c/576236/ | |
| 20:06:24 | efried | mriedem: ack, thx | |
| 20:06:26 | mriedem | smcginnis: we only save 10 days of logs | |
| 20:06:42 | smcginnis | Well poop | |
| 20:55:46 | openstackgerrit | Eric Fried proposed openstack/nova master: Fail heal_allocations if placement is borked https://review.openstack.org/597237 | |
| 20:55:55 | efried | mriedem: ^ | |
| 20:59:00 | mriedem | mmm, bork | |
| 21:04:51 | mriedem | efried: i need a retry decorator, do you remember any pros of retrying.retry over oslo's RetryDecorator? | |
| 21:05:08 | efried | mriedem: yeah, lemme context switch... | |
| 21:05:51 | efried | mriedem: I think the main advantage of retrying is: no threading. | |
| 21:06:14 | efried | But RetryDecorator has a simpler/better-documented interface | |
| 21:10:13 | mriedem | ok that's about what i remembered | |
| 21:10:21 | mriedem | during the RT._update thing | |
| 21:15:54 | melwitt | mriedem: I've been using a script for counting blueprint stuff for future charting, if you wouldn't mind reviewing https://review.openstack.org/592628 (second patch in series) | |
| 21:28:06 | mriedem | +2 | |
| 21:28:59 | efried | mriedem: Responded to your questions on https://review.openstack.org/#/c/576236/ | |
| 21:29:22 | efried | mriedem: Let me know if you want anything acted on immediately, or if fups are okay. | |
| 21:29:35 | mriedem | mmm f ups | |
| 21:29:39 | mriedem | yeah was just opening that | |
| 21:29:57 | mriedem | there is also a strange small human in my house now since school is out so i'll be potentially distracted | |
| 21:30:58 | efried | mriedem: No huge hurry, I think I'm probably going to need to knock off for the evening soon. | |
| 21:31:24 | melwitt | mriedem: thanks. the counting blueprint script is the second patch stacked on top of that one, though. pls | |
| 21:31:52 | mriedem | oh right... | |
| 21:40:27 | mriedem | efried: +2 on that | |
| 21:40:31 | mriedem | https://review.openstack.org/#/c/576236/ that is | |
| 21:40:41 | efried | mriedem: Cool, thx | |
| 21:40:41 | mriedem | jaypipes was +2 so i suppose i could +W | |
| 21:40:47 | efried | yahsure | |
| 21:40:50 | mriedem | jaypipes: do you want to +W https://review.openstack.org/#/c/576236/ ? | |
| 21:40:56 | mriedem | or happy to have me proxy | |
| 21:40:59 | efried | that would put the whole pile in the gate. | |
| 21:41:11 | jaypipes | mriedem: done | |
| 21:42:02 | efried | woot | |
| 21:45:10 | openstackgerrit | Eric Fried proposed openstack/nova master: Name arguments to _get_provider_ids_matching https://review.openstack.org/597291 | |
| 22:24:20 | openstackgerrit | Eric Fried proposed openstack/nova master: Other host allocs may appear in gafpt during evac https://review.openstack.org/597301 | |
| 22:37:54 | openstackgerrit | Eric Fried proposed openstack/nova master: Mention (unused) RP generation in POST /allocs/{c} https://review.openstack.org/597304 | |
| 22:54:07 | openstackgerrit | melanie witt proposed openstack/nova-specs master: Propose configurable maximum number of volumes to attach https://review.openstack.org/597306 | |
| 23:34:32 | openstackgerrit | Merged openstack/nova master: Don't use '_TransactionContextManager._async' https://review.openstack.org/597173 | |
| 23:34:38 | openstackgerrit | Merged openstack/nova master: Revert "Don't use '_TransactionContextManager._async'" https://review.openstack.org/597174 | |
| 23:40:32 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106 | |
| 23:40:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110 | |
| 23:40:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (6) https://review.openstack.org/574113 | |
| 23:41:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (7) https://review.openstack.org/574974 | |
| 23:41:24 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311 | |
| 23:41:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.openstack.org/575581 | |
| 23:43:51 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.openstack.org/576017 | |
| 23:43:59 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.openstack.org/576018 | |
| 23:44:23 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.openstack.org/576019 | |
| 23:44:52 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.openstack.org/576020 | |
| 23:45:15 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.openstack.org/576027 | |
| 23:45:32 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.openstack.org/576031 | |
| 23:45:54 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299 | |