| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-22 | |||
| 14:44:07 | aspiers | yup | |
| 14:44:25 | aspiers | will do now | |
| 14:45:18 | dansmith | mriedem: done | |
| 14:45:27 | mdbooth | aspiers: If you find that a patch which isn't on the bottom gets +2s you can always rebase it to the bottom. If it's a clean rebase you'll retain the +2s and it can merge. | |
| 14:45:40 | aspiers | mdbooth: nice tip, thanks | |
| 14:45:52 | mdbooth | aspiers: It's not great, but it's kinda ok and doesn't usually come up that much. | |
| 14:46:10 | aspiers | I suspect Gerrit 2.16 will handle this better, as I mentioned the other day | |
| 14:46:13 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Don't emit warning when ironic properties are zero https://review.openstack.org/608611 | |
| 14:46:29 | aspiers | Oh, that wasn't in this channel | |
| 14:46:48 | lyaaaaarwood | mdbooth: pretty sure you can migrate between two backends on the same host | |
| 14:47:02 | lyaaaaarwood | mdbooth: that's what I've been testing locally at least | |
| 14:47:09 | lyaaaaarwood | mdbooth: they don't need to be different c-vols | |
| 14:47:50 | aspiers | Ah, it was :-) mdbooth: in case you missed it: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-02-12.log.html#t2019-02-12T22:08:56 | |
| 14:50:41 | lyaaaaarwood | mdbooth: https://developer.openstack.org/api-ref/block-storage/v3/index.html?expanded=migrate-a-volume-detail#migrate-a-volume - when the docs talk about migrating to a host they mean hostname@backend where hostname can be the same host still. | |
| 14:52:24 | mdbooth | lyaaaaarwood: Isn't that retype? | |
| 14:53:07 | kashyap | mriedem: Looking ... my IRC proxy dropped me off of Freenode | |
| 14:53:54 | kashyap | mriedem: Did you file a bug yet? Or shall I? | |
| 14:54:53 | lyaaaaarwood | mdbooth: https://developer.openstack.org/api-ref/block-storage/v3/index.html?expanded=migrate-a-volume-detail,complete-migration-of-a-volume-detail,retype-a-volume-detail#retype-a-volume - that's a totally different API | |
| 14:55:08 | lyaaaaarwood | mdbooth: where you can do the same thing, just with types that have backends defined in them. | |
| 14:55:20 | lyaaaaarwood | mdbooth: and again they take the form of hostname@backend | |
| 14:55:29 | lyaaaaarwood | mdbooth: so you can migrate between backends on the same host | |
| 14:55:38 | lyaaaaarwood | ^ sorry retype | |
| 14:56:37 | openstackgerrit | Adam Spiers proposed openstack/nova master: Change LibvirtDriver.capabilities to an instance variable https://review.openstack.org/638677 | |
| 14:56:48 | mdbooth | lyaaaaarwood: So retype and migrate are literally identical operations, just with different semantics for specifying the source and dest? | |
| 14:56:58 | mriedem | kashyap: https://bugs.launchpad.net/nova/+bug/1817324 | |
| 14:56:59 | openstack | Launchpad bug 1817324 in OpenStack Compute (nova) "Intermittent "Failed to start libvirt guest: libvirt.libvirtError: monitor socket did not show up: No such file or directory" failures in the gate" [Undecided,Confirmed] | |
| 14:57:08 | lyaaaaarwood | mdbooth: yeah pretty much | |
| 14:57:15 | kashyap | Thanks | |
| 14:57:37 | mdbooth | lyaaaaarwood: I don't understand, tbh. | |
| 14:58:10 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add detection of SEV support from QEMU/AMD-SP/libvirt on AMD hosts https://review.openstack.org/633855 | |
| 14:59:42 | kashyap | mriedem: On your comment in the bug; for that failure, the interesting stuff won't be in the QEMU guest log; it would be in the libvirt <-> QEMU interactions log (which the Gate captures) | |
| 14:59:59 | lyaaaaarwood | mdbooth: with migrate you provide the host (hostname@backend) you want to migrate to. With retype you provide a type you'd like to migrate to that can optionally contain a host (hostname@backend) extra spec that moves the volume | |
| 15:00:02 | mriedem | kashyap: i looked in the libvirtd log and didn't really see anything interesting | |
| 15:00:20 | mriedem | kashyap: besides stuff like it created the monitor for that instance and events weren't getting handled, something like that | |
| 15:01:23 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add new "supports_amd_sev" capability to libvirt driver https://review.openstack.org/638680 | |
| 15:01:36 | kashyap | Nod; /me still downloading the log | |
| 15:09:12 | frickler | mriedem: anything I can do to help with https://bugs.launchpad.net/nova/+bug/1815082 ? | |
| 15:09:13 | openstack | Launchpad bug 1815082 in OpenStack Compute (nova) ""DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 15:09:30 | frickler | I'm also wondering whether this may be a duplicate actually https://bugs.launchpad.net/devstack/+bug/1814016 | |
| 15:09:31 | openstack | Launchpad bug 1814016 in devstack "Subnode required CELLSV2_SETUP=superconductor to be explicitly set" [Undecided,New] | |
| 15:15:46 | mriedem | frickler: ah i forgot about that | |
| 15:17:35 | frickler | mriedem: o.k., I'm assuming I did my part of the job, then. ;) | |
| 15:18:19 | mriedem | well n-api-meta is hitting conductor http://logs.openstack.org/77/635577/2/check/tempest-full/152e5bc/controller/logs/screen-n-api-meta.txt.gz#_Feb_08_16_53_34_572980 | |
| 15:18:32 | mriedem | i think | |
| 15:18:41 | mriedem | but it seems that the conductor it's hitting isn't up yet | |
| 15:20:15 | mriedem | we start n-api-meta here | |
| 15:20:15 | mriedem | http://logs.openstack.org/77/635577/2/check/tempest-full/152e5bc/controller/logs/devstacklog.txt.gz#_2019-02-08_16_52_32_215 | |
| 15:20:19 | frickler | http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n1096 | |
| 15:20:37 | frickler | that's in start_nova_rest, way before _conductor, yes | |
| 15:21:04 | mriedem | yeah n-super-cond is started here: | |
| 15:21:04 | mriedem | http://logs.openstack.org/77/635577/2/check/tempest-full/152e5bc/controller/logs/devstacklog.txt.gz#_2019-02-08_16_52_36_721 | |
| 15:21:44 | frickler | probably n-api-meta should idle a bit and retry | |
| 15:22:07 | frickler | even if we fix the order in devstack, we can't assume deployers will fine tune this everywhere | |
| 15:22:37 | mriedem | start_nova_rest is kind of weird - it starts nova-network as well which has always needed conductor to start first | |
| 15:22:39 | kashyap | mriedem: Oops: I think it is due to two 'pty' consoles getting reated...see lines 87-97 (from the n-cpu log): http://paste.openstack.org/show/745736/ | |
| 15:23:30 | mriedem | kashyap: would that be due to your recent version bump change? | |
| 15:23:34 | mriedem | which messed with the console code? | |
| 15:23:46 | kashyap | Yes, very much likely :-( | |
| 15:23:55 | kashyap | I fixed the s390x case, and I didn't see this at all in normal runs | |
| 15:23:56 | kashyap | Sigh | |
| 15:24:03 | kashyap | This seems to be 'isa-serial'-related | |
| 15:24:15 | mriedem | something something risk of doing this at the end of a release... | |
| 15:28:32 | kashyap | Yeah, I know what you mean. I made at least 3 folks to review, including the last person who reworked that code | |
| 15:28:55 | kashyap | I'll work towards fixing it as quickly as I can. | |
| 15:31:49 | mriedem | frickler: i think we're hitting the wrong conductor maybe | |
| 15:32:10 | mriedem | frickler: conductor is effectively started at 16:52:40 | |
| 15:32:14 | mriedem | n-super-cond that is | |
| 15:32:25 | mriedem | n-api-meta times out at 16:53:34 | |
| 15:32:38 | mriedem | note there is a 60 second rpc_response_timeout on that db query | |
| 15:33:46 | mriedem | problem is i'm not exactly sure which conductor n-api-meta is trying to hit | |
| 15:35:53 | mriedem | need to push a change to dump the config on startup before trying to hit the db | |
| 15:36:43 | kashyap | mriedem: Okay, one thing eliminated: that XML bit is a red-herring; it's not the cause. | |
| 15:36:51 | kashyap | (And the patch for consoles is still correct.) | |
| 15:37:36 | kashyap | (The seemingly two consoles is due to a silly backward compantbiility thing in libvirt) | |
| 15:40:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Set the conductor indirection API when running nova-metadata under uwsgi https://review.openstack.org/635577 | |
| 15:40:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Dump config options on wsgi startup earlier https://review.openstack.org/638691 | |
| 15:43:19 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: tox: Don't write byte code (maybe) https://review.openstack.org/636919 | |
| 15:47:49 | kashyap | mriedem: Do you know if these Ubuntu machines are using AppArmour? | |
| 15:48:38 | sean-k-mooney | kashyap: its installed by default so proably | |
| 15:49:15 | mriedem | kashyap: that's a question for infra | |
| 15:51:03 | kashyap | Okay | |
| 15:51:30 | kashyap | Talking to a libvirt dev, the core problem is QEMU failed to start, and consequently failed to create the socket. | |
| 15:51:36 | kashyap | Now trying to determine why | |
| 15:57:56 | kashyap | sean-k-mooney: Thanks | |
| 16:10:51 | giblet | mriedem: regarding your comments in https://review.openstack.org/#/c/569459/67/nova/network/neutronv2/api.py | |
| 16:11:04 | giblet | mriedem: i did not consider live-migration or other server move operation there | |
| 16:11:39 | giblet | mriedem: I think _update_port_binding_for_instance() needs to change when we enable server move operations | |
| 16:12:04 | giblet | mriedem: the RP uuid fulfilling a port's allocation will definitly change when we move the server to another host | |
| 16:18:56 | kashyap | mriedem: BTW, so this kind of completely silent failure is typically caused due to SELinux/AppArmour denials | |
| 16:19:11 | kashyap | But that's rather difficult to debug in OpenStack envs... | |
| 16:19:32 | kashyap | I'd be interested if this is reproducible at all -- should get to know it is by Monday | |
| 16:21:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log why rescheduling is disabled https://review.openstack.org/638699 | |
| 16:21:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove misleading comment from _move_operation_alloc_request() https://review.openstack.org/638700 | |
| 16:21:15 | mriedem | leakypipes: dansmith: some fun from the past ^ | |
| 16:26:42 | sean-k-mooney | kashyap: wouldnt that show up in the audit.log | |
| 16:27:31 | kashyap | sean-k-mooney: No idea what AppArmour does; let me see, though | |
| 16:27:37 | kashyap | I know how to troubleshoot SELinuz | |
| 16:28:32 | sean-k-mooney | both used to show up in demesg then they moved to journalctl but were also loged to the audit.log under /var/log i think | |
| 16:31:01 | sean-k-mooney | this is also a kernel bug in ubuntu 18.04 that causes nested virt to not work but if that iss the issue the core dump in dmesg telegraphfs the issue rather quirckly. | |
| 16:32:09 | kashyap | sean-k-mooney: Where is the audit.log here: http://logs.openstack.org/48/631948/9/check/tempest-full-py3/e2ae3fb/controller/logs/ | |