Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-10
20:05:23 ildikov and it uses the default for reboot as it's supposed to
20:06:32 ildikov can anything in libvirt call that?
20:06:47 mriedem there are a couple of places but they are during snapshot
20:08:28 openstackgerrit Merged openstack/nova master: use already loaded BDM in instance.create https://review.openstack.org/483969
20:08:30 ildikov hmm, weird, as there's nothing around that update call, but brick doing its things...
20:09:53 openstackgerrit Merged openstack/nova master: Fix sending legacy instance.update notification https://review.openstack.org/510603
20:11:06 jgriffith_ mriedem ildikov http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a079b9/logs/screen-c-api.txt.gz?#_Oct_10_12_53_01_672274
20:12:04 ildikov jgriffith_: I think that's 'begin_detaching'
20:12:11 jgriffith_ yes it is
20:12:18 ildikov jgriffith_: which we left untouched
20:12:28 jgriffith_ "we" ?
20:12:35 ildikov so that call is normal
20:12:53 ildikov jgriffith_: I kept you in the loop with that :)
20:13:37 ildikov jgriffith_: I mean to keep that in the flow as we don't want parallel detaches happening as we don't have anything to change the volume or whatever state to 'detaching'
20:15:08 openstackgerrit Eric Fried proposed openstack/nova master: DNM: CI check https://review.openstack.org/511006
20:15:54 mriedem i'm not really following what begin_detaching has to do with this
20:16:03 mriedem that's the symptom
20:16:06 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
20:16:10 mriedem looking at the tempest logs, the volume is in-use here http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_909920
20:16:34 mriedem then we reboot the server, and i see the phantom attachment_update call: http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_02_05_685303
20:17:03 ildikov jgriffith_: I know, we are not supposed to keep old flow stuff neither copy it to the new one, it seemed a reasonable idea to keep this for now back at the time
20:17:11 mriedem and the failed detach here http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_908565
20:17:38 mriedem and http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_909920 shows the volume went from in-use to attaching
20:18:33 ildikov mriedem: yeah, that last bit is the consequence of update
20:19:18 mriedem ildikov: if the new refresh_connection_info code thinks that calling attachment_update means we're attaching, then that's a bug
20:19:24 mriedem i thought we talked about this at the ptg
20:20:56 ildikov mriedem: about refresh?
20:21:05 mriedem yeah i'm looking up the etherpad
20:21:54 jgriffith_ ahh, I see; never did the attachment-complete on the reboot
20:22:14 mriedem there shouldn't be an attachment_update on the reboot
20:22:15 jgriffith_ mriedem and yes, you're correct
20:22:17 mriedem i don't know where it's coming from
20:24:10 ildikov mriedem: regardless of how refresh is behaving, there still doesn't seem to be a logical path to that attachment_update call :/
20:25:58 mriedem L269 https://etherpad.openstack.org/p/cinder-ptg-queens
20:26:02 mriedem apparently my notes were removed...
20:26:21 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/511021
20:28:00 ildikov mriedem: https://etherpad.openstack.org/p/cinder-ptg-queens-thursday-notes
20:28:09 ildikov L121
20:28:41 mriedem whew
20:28:43 mriedem thanks
20:29:04 mriedem "With the new style Cinder API, does nova.virt.block_device.DriverVolumeBlockDevice.refresh_connection_info change? Do we do an attachment_show or do we delete the old attachment and create a new one (or does that only happen on a HARD reboot - if we make that change)?"
20:29:36 mriedem "if bdm.attachment_id then we should fetch connection info from Cinder (on hard reboot) What I heard is this actually needs to be an attachment_delete/create swap thing because that's what gives us the connection_info from the storage backend. Although I wonder if attachment_update would refresh the connection_info on the cinder side...since attachment_update == os-initialize_connection."
20:33:53 ildikov not sure that's fully equal
20:34:23 ildikov I'm always open to be corrected by jgriffith_ :)
20:36:17 ildikov but with reboot we call the _get_instance_block_device_info with default on refresh, which is False
20:37:53 mriedem ildikov: what i'm going to do is just push a debug patch on top of john's patch which will dump a stacktrace if we call attachment_update while rebooting an instance
20:38:12 ildikov mriedem: sounds good, thank you
20:38:13 mriedem i've spent way too much time looking at logs today
20:38:23 ildikov mriedem: sorry :(
20:38:32 mriedem nah it's fine,
20:38:38 mriedem i just need to let the computer do some work
20:38:51 ildikov mriedem: I swear I just rebased the thing and bumped the service version :S
20:39:14 ildikov mriedem: +1 on the last comment :)
20:42:27 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146
20:48:35 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
20:50:01 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Fail if we try to update volume connections during reboot https://review.openstack.org/511031
20:54:51 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/511021
20:56:56 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/511035
21:08:42 cdent edleafe: in the commit message on https://review.openstack.org/#/c/510625/4 there are some questions about possible todos, did you have an opinion on those?
21:09:34 edleafe cdent: you mean about including the project_id/user_id in the GET?
21:09:44 cdent yeah
21:09:58 edleafe I don't see why we shouldn't
21:10:38 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/511021
21:11:10 coreycb hi all, cells v2 question
21:11:38 cdent edleafe: does that mean I should put a -1 or -W or a something on there?
21:12:40 coreycb i'm hitting a window where after 'nova-manage cells_v2 list_cells' starts returning the cell i'm interested in, mapping hosts or instances for that cell fails during the window until the database is finished. is that expected?
21:12:44 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/511035
21:13:58 mriedem coreycb: until the database is finished with what?
21:16:29 openstackgerrit Chris Dent proposed openstack/nova master: Move project_id and user_id to Allocation object https://review.openstack.org/500410
21:16:30 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Allow _set_allocations to delete allocations https://review.openstack.org/501051
21:16:31 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Limit number of attempts to delete allocations https://review.openstack.org/507224
21:16:32 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Enable PUT dict to /allocations/{consumer_uuid} https://review.openstack.org/510625
21:16:33 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Document the changes to PUT /allocations/{consumer_uuid} https://review.openstack.org/510626
21:16:34 openstackgerrit Chris Dent proposed openstack/nova master: [placement] POST /allocations to set allocations for >1 consumers https://review.openstack.org/500073
21:17:37 edleafe cdent: You mean so they can be in the same microversion? Meh, they're cheap. You can do it in a follow-up patch if you like
21:18:24 coreycb mriedem: it seems to still be initializing the DB from prior nova-manage commands. here's some output of 'list_cells' looping and then running 'discover_hosts' once the cell starts showing up in list_cells output: https://launchpadlibrarian.net/339984742/printscreen
21:18:44 cdent edleafe: no, not worried about the microversion issue, just that somewhere it needs to be marked as we’re not done here. If it is to be done, I’d be inclined to do it in a prior patch.
21:18:53 cdent i’ll leave a comment-to-self
21:19:09 coreycb mriedem: search for 'cell1'
21:20:27 mriedem coreycb: Table 'nova.compute_nodes' doesn't exist
21:20:40 mriedem you can't run discover_hosts or map_instances unti you've synced the dbs on the cells
21:20:55 mriedem meaning, you need to run nova-manage db sync on each cell database
21:21:27 mriedem https://docs.openstack.org/nova/pike/user/cells.html#setup-of-cells-v2
21:22:09 mriedem "At this point, the API database can now find the cell database, and further commands will attempt to look inside. If this is a completely fresh database (such as if you’re adding a cell, or if this is a new deployment), then you will need to run nova-manage db sync on it to initialize the schema."
21:22:16 coreycb mriedem: i think we do that, we basically follow the order of what's in doc/source/cells.rst
21:22:35 coreycb mriedem: it's just a small window and then you'll see at the end of that output discover_hosts works.
21:23:37 mriedem coreycb: so there is a separate process that's calling create_cell and doing the db sync on each cell?
21:28:31 coreycb mriedem: no it's same process, but we don't target the cell on the db sync if that's possible
21:29:15 mriedem nova-manage db sync will sync the current cell db (so cell1 if you're running it on a nova.conf with [database] configured for cell1) and it will query the nova_api database looking for cell0 and sync that as well
21:29:30 mriedem that has to happen before you start discovering hosts in cell1
21:30:53 coreycb mriedem: ok. yes we're doing that. basically we do steps 1-6 of the document you linked, and then if i run list_cells and discover_hosts in a loop, there will be a window (i've not timed it, maybe 30 seconds?) where list_cells returns 'cell1' but discover_hosts fails. then after eventually discover_hosts works.
21:31:27 coreycb mriedem: just curious really if that's normal behavior
21:33:26 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/511021
21:33:33 mriedem well i can tell from that dump that you're trying to discover_hosts before cell1 db schema is fully sync'ed
21:33:45 mriedem because
21:33:45 mriedem ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.compute_nodes' doesn't exist")
21:34:21 coreycb mriedem: right. my impression from that is that nova-manage commands aren't necessarily finished when they return. but i'm guessing.
21:34:59 mriedem db sync is synchronoous
21:35:36 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/511035
21:36:38 coreycb mriedem: ok that's good to know. if that's the case it must be something on my end then.

Earlier   Later