| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-28 | |||
| 22:19:32 | openstackgerrit | Eric Fried proposed openstack/nova master: FUPs: ReportClient traffic series https://review.openstack.org/633609 | |
| 22:26:48 | efried | jaypipes, gibi, edleafe: your fups have arrived ^ | |
| 22:28:27 | edleafe | efried: cool, will look soon | |
| 22:52:14 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells https://review.openstack.org/591657 | |
| 22:58:07 | openstackgerrit | Hang Yang proposed openstack/nova master: Fix port dns_name reset https://review.openstack.org/631684 | |
| 23:11:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix deps for api-samples tox env https://review.openstack.org/633620 | |
| 23:11:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add microversion to expose virtual device tags https://review.openstack.org/631948 | |
| 23:11:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make VolumeAttachmentsSampleV249 test other methods https://review.openstack.org/633621 | |
| #openstack-nova - 2019-01-29 | |||
| 00:16:11 | openstackgerrit | Eric Fried proposed openstack/nova master: FUPs: ReportClient traffic series https://review.openstack.org/633609 | |
| 00:50:46 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to spec obj. https://review.openstack.org/631243 | |
| 04:24:52 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to spec obj. https://review.openstack.org/631243 | |
| 04:24:52 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.openstack.org/631242 | |
| 04:27:15 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to spec obj. https://review.openstack.org/631243 | |
| 04:42:29 | openstackgerrit | Hesam Chobanlou proposed openstack/nova master: docs: ComputeDriver.update_provider_tree in nova https://review.openstack.org/633375 | |
| 06:00:06 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove deprecated 'os-flavor-manage' policy https://review.openstack.org/633656 | |
| 06:10:15 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix string interpolations in logging calls https://review.openstack.org/633657 | |
| 07:41:59 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove deprecated 'flavors' policy https://review.openstack.org/633664 | |
| 07:42:02 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to spec obj. https://review.openstack.org/631243 | |
| 07:42:02 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.openstack.org/631242 | |
| 07:42:03 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.openstack.org/631244 | |
| 08:06:52 | openstackgerrit | Yongli He proposed openstack/nova master: Adds the server group info into show server detail API. https://review.openstack.org/621474 | |
| 08:15:19 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/627765 | |
| 08:36:53 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove deprecated 'os-server-groups' policy https://review.openstack.org/633672 | |
| 09:02:21 | kashyap | gibi: Morning, are others seeing these timeouts too? https://review.openstack.org/#/c/630980/ | |
| 09:02:38 | kashyap | It's been like that for more than 10 days | |
| 09:30:21 | jangutter | kashyap: you can get a quick "feel" by checking http://zuul.openstack.org/builds?job_name=tempest-full&pipeline=gate and there does indeed seem to be a few hits. | |
| 09:30:32 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix a missing policy in test policy data https://review.openstack.org/633686 | |
| 09:30:59 | kashyap | jangutter: Hi, a few hits of what? | |
| 09:31:58 | jangutter | kashyap: TIMED_OUT on tempest-full builds on the gate. Obviously this is highly unscientific, but a quick way to check. | |
| 09:37:34 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix a missing policy in test policy data https://review.openstack.org/633686 | |
| 09:38:43 | jangutter | kashyap: oh wow, that review seems to love hitting random gate failures. | |
| 09:38:53 | kashyap | Yeah :-( | |
| 09:39:04 | kashyap | Anyway to fix that? | |
| 09:43:09 | jangutter | kashyap: that's down to bugfixing, unfortunately. Digging into the logs might help to try to find out if it's something you can spot. | |
| 09:43:33 | kashyap | jangutter: Will do. But the last time I did, nothing was related to my patch :-( | |
| 09:43:56 | jangutter | kashyap: yep, I can almost 100% guarantee that there isn't. | |
| 09:44:10 | jangutter | kashyap: (a fix related to your patch, I mean). | |
| 09:44:32 | kashyap | I don't see any failures here either: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/testr_results.html.gz | |
| 09:45:35 | jangutter | kashyap: I think devstack took too long to spin up. | |
| 09:50:14 | jangutter | kashyap: If you look at http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/ara-report/ <--- it took 30 minutes for the run-devstack task. | |
| 09:50:54 | jangutter | kashyap: some of these failures require a skill level that waay beyond my ability to comprehend. | |
| 09:52:49 | kashyap | Hmm | |
| 09:52:56 | kashyap | Thanks for looking; I'll check w/ the -infra folks | |
| 09:56:40 | jangutter | kashyap: yeah, digging into that ara-report, there's a 30 minute timeout on tasks, and the task went too long. Could be that the VM was running under contention or something so there's a very high chance a recheck will succeed. | |
| 09:56:58 | kashyap | Alright, we've had about 6 re-checks | |
| 09:57:13 | kashyap | Another one is in progress; let's see. | |
| 11:34:15 | tinwood | Morning. Is there anyone who could take a look at https://review.openstack.org/#/c/633279/ for me; I needed to add the LXD bit, but not sure if I should have version bumped. Many thanks! | |
| 11:49:49 | gibi | kashyap: I didn't see such tempest timeout recently but not long ago we had problems with tempest slowness and moved some of the tests to tempest-slow | |
| 11:51:15 | ioni | hello guys | |
| 11:51:34 | gibi | kashyap: if you can dig up what are the slowest test in those runs that timed out then we can move some more test to tempest-slow | |
| 11:51:41 | ioni | what's the right way to disable out of date images ? | |
| 11:51:53 | ioni | --deactivate or --private or other way? | |
| 11:52:36 | ioni | because now i'm using --deactive and if an instance that is using this image wants to resize, nova returnes and error that the project doesn't have permission to that specific image | |
| 11:52:40 | ioni | that is now in error state | |
| 11:52:57 | ioni | i'm doing something wrong or is a bug? | |
| 11:55:11 | ioni | the error log: https://paste.xinu.at/b6ZsNA/ | |
| 12:56:36 | jaosorior | kashyap: Hello! Are you the right person to ask a libvirtd related question? | |
| 13:02:03 | kashyap | gibi: Give me a few; check out this: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/ara-report/ | |
| 13:02:23 | kashyap | gibi: Pointed out by jangutter earlier; where the 'run-devstack' task was taking 30 minutes | |
| 13:02:33 | kashyap | jaosorior: Hi, you can you ask, I can try :-) | |
| 13:03:18 | jaosorior | kashyap: I'm trying to reload libvirtd, given that it's certificate got rotated. in the man pages it says SIGHUP should do that... but it seems it doesn't... wanted to know if this is a bug, or if there's some other mechanism I could/should use instead. | |
| 13:06:54 | kashyap | jaosorior: Hmm, interesting problem | |
| 13:07:26 | kashyap | jaosorior: Can you afford to do an explicit stop+start of the daemon? | |
| 13:07:58 | kashyap | If so, does that pick up the new certs? | |
| 13:08:48 | jaosorior | kashyap: would that disrupt instances running on the host? | |
| 13:08:57 | kashyap | jaosorior: Good news: no. | |
| 13:09:07 | kashyap | jaosorior: Hang on, though. | |
| 13:09:13 | jaosorior | kashyap: what about connections from nova? | |
| 13:09:16 | kashyap | First, is that a production instance? | |
| 13:09:26 | jaosorior | yes | |
| 13:09:35 | jaosorior | well... not where I'm testing all this stuff :D | |
| 13:09:37 | kashyap | jaosorior: But know that: restarting `libvirtd` will _not_ bring your instances down. | |
| 13:09:40 | jaosorior | but that's the goal, yes | |
| 13:11:24 | kashyap | jaosorior: BTW, the SIGHUP config is ill-defined in the `libvirtd` man page :-( | |
| 13:11:39 | kashyap | It says: "On receipt of SIGHUP libvirtd will reload its configuration." | |
| 13:11:46 | jaosorior | right | |
| 13:11:50 | kashyap | But doesn't expand what is "its configuration". | |
| 13:12:02 | kashyap | I'll get that sorted w/ the libvirt upstream to expand and elaborate. | |
| 13:12:03 | jaosorior | TLS certs/keys are configuration IMO :D | |
| 13:12:59 | kashyap | jaosorior: Okay, so you're using TLS. | |
| 13:13:42 | kashyap | And you want to pick up renewed certificates. Then problem is not libvirtd, I'm afraid — QEMU is the real blocker | |
| 13:14:27 | jaosorior | what are my options? | |
| 13:14:31 | kashyap | Because it (QEMU) can't reload certificates on the fly. (And also QEMU can't just be "restarted") | |
| 13:14:51 | kashyap | Let me check with libvirt folks. | |
| 13:15:37 | jaosorior | kashyap: thanks! | |
| 13:16:20 | kashyap | jaosorior: So, your only viable option is a bit heavy: live migration of the QEMU instance(s). | |
| 13:16:21 | kashyap | (I.e. Nova instances) | |
| 13:16:48 | ioni | where the certs are used? | |
| 13:19:21 | kashyap | ioni: For TLS for various Nova services, I presume | |
| 13:19:47 | kashyap | jaosorior: Related, recently we merged this: http://git.openstack.org/cgit/openstack/nova/tree/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst | |
| 13:20:04 | kashyap | There is a nicer rendered version | |
| 13:20:53 | kashyap | jaosorior: http://logs.openstack.org/83/631283/3/check/openstack-tox-docs/615979c/html/admin/secure-live-migration-with-qemu-native-tls.html | |
| 13:22:08 | jaosorior | kashyap: I'm a little confused though. the TLS implementation for connections to libvirtd is not the same one then as the one for qemu, right? | |
| 13:22:27 | kashyap | (But the above URL doesn't solve the existing problem, though. It is "future goodness", when you upgrade.) | |
| 13:23:20 | kashyap | jaosorior: Are you using the 'live_migration_tunnelled' Nova config? | |
| 13:24:05 | kashyap | jaosorior: They're somewhat connected: when you use TLS with libvirt, it is passing the TLS creds under the hood to QEMU. | |
| 13:24:18 | jaosorior | kashyap: we are, yeah | |
| 13:24:19 | kashyap | jaosorior: I can tell more details, if you post your QEMU command-line of your Nova instance (from the TLS env) | |
| 13:24:30 | jaosorior | kashyap: sure | |
| 13:24:55 | kashyap | I need to grab a quick bite (not going to eat in front of the computer, as a rule). Be back in 15 mins | |