| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-04 | |||
| 13:38:31 | pipesinpain | jmccarthy: you are using volume-backed instances, yes? | |
| 13:38:41 | pipesinpain | jmccarthy: i.e. boot-from-volume | |
| 13:39:44 | jmccarthy | I have to check what horizon says, spawning cirros with cinder, so volume backed yep | |
| 13:40:11 | pipesinpain | jmccarthy: right. so that is the behaviour by design. | |
| 13:40:38 | pipesinpain | jmccarthy: maciej had added that code to ensure that the volume wasn't removed when resizing. | |
| 13:40:57 | pipesinpain | jmccarthy: for boot-from-volume instances, you don't want to delete the original root volume, clearly :) | |
| 13:41:28 | pipesinpain | jmccarthy: I'm wondering if this is because of some thin/sparse copy-on-write stuff that Docker is doing maybe.. | |
| 13:42:05 | jmccarthy | pipesinpain: The volume itself is in cinder and seems safe enough in my case ? It's this disk.info that shows up on host that is throwing me off | |
| 13:42:22 | pipesinpain | jmccarthy: but in all honesty, I'm pretty much the worst person to ask on this :) a) I have little knowledge of the block device layer, b) I don't use (or support) boot-from-volume, and c) I don't use resize ;) | |
| 13:42:55 | pipesinpain | jmccarthy: good people to hit up are mriedem when he's online and maybe mdbooth | |
| 13:43:08 | jmccarthy | pipesinpain: Oh ok I thought you might know, since my disk.info issue sounds a lot like a bug you fixed up | |
| 13:43:43 | jmccarthy | pippesinpain: Ok I'll ask around - I'm just not sure why it would leave this file on the host afterwards, seems odd | |
| 13:43:44 | pipesinpain | jmccarthy: that was more melwitt (soon to be jgwentworth) that fixed that bug :) | |
| 13:43:48 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove support for monitoring Intel CMT `perf` events https://review.openstack.org/565242 | |
| 13:43:57 | jmccarthy | pipesinpain: Ok ! :) Sorry for the noise | |
| 13:44:05 | pipesinpain | jmccarthy: no problem at all, man :) | |
| 13:44:38 | jmccarthy | Any of those people sitting quielty by ? ;) | |
| 13:46:14 | pipesinpain | jmccarthy: melwitt will be up in the next hour or so. mriedem will be by soon I imagine (PST and CST respectively) | |
| 13:46:36 | jmccarthy | In horizon, if it boots from image and you create a new volume at the same time in persistent storage for it, that not boot-from-volume strictly speaking, correct ? | |
| 13:46:44 | pipesinpain | jmccarthy: mdbooth is in GMT so should be around (he's likely seen "boot from volume" and run screaming for the hills) :P | |
| 13:47:04 | pipesinpain | jmccarthy: no, that's not boot-from-volume. | |
| 13:47:26 | jmccarthy | pipesinpain: That is a volume backed instance, but booted off image - does that have a special name ? | |
| 13:47:35 | pipesinpain | jmccarthy: boot-from-volume is when you supply a prebuilt bootable volume as your root disk (disk 0) for the instance | |
| 13:47:50 | pipesinpain | jmccarthy: that's a normal instance, then | |
| 13:47:59 | jmccarthy | pipesinpain: I am using volumes, but nope, not boot from vol, ok normal instances | |
| 13:48:04 | pipesinpain | jmccarthy: you mean you just added a non-bootable data volume to the instance, right? | |
| 13:48:17 | wolverineav | hi, while working on a neutron plugin, i ran into a weird situation where nova sends multiple requests for bind_port on different hosts, before neutron has had a chance to complete the first request. i’m guessing this would be by design, since I don’t see error or warning about the first request failing. in case anyone would know where the code path for this resides, that’d be great! | |
| 13:48:24 | pipesinpain | jmccarthy: good man, yes, that's not boot-from-volume :) | |
| 13:48:54 | jmccarthy | pipesinpain: Well not manually ? Horizon just does it .. "Instance source is the template used to create an instance. You can use an image, a snapshot of an instance (image snapshot), a volume or a volume snapshot (if enabled). You can also choose to use persistent storage by creating a new volume." | |
| 13:49:04 | pipesinpain | wolverineav: ironic? | |
| 13:49:26 | wolverineav | no, instance creation in overcloud | |
| 13:49:30 | mriedem | jmccarthy: nova show the instance on the CLI, if the image_ref is "", it's volume-backed (boot from volume) | |
| 13:49:37 | pipesinpain | jmccarthy: right. and you chose the "persistent storage by creating a new volume", yes? | |
| 13:49:50 | wolverineav | pipesinpain: no, instance creation in overcloud | |
| 13:50:02 | jmccarthy | pipesinpain: Mine just says 'create new volume' but yep | |
| 13:50:54 | pipesinpain | wolverineav: multiple requests to bind the port to different hosts for the same instance? :( | |
| 13:51:12 | jmccarthy | pipesinpain: Afaik this creates an instance the lives solely in the volume like, and subsequently will boot from there | |
| 13:52:15 | jmccarthy | pipesinpain: 'pipesinpain' sounds painful | |
| 13:56:47 | pipesinpain | jmccarthy: no. unless I'm mistaken (very much a possibility), "create a volume" just means the "You can also choose to use persistent storage by creating a new volume." option. In other words, it creates a data volume (non-bootable) for the instance to use. | |
| 13:56:58 | wolverineav | pipesinpain: sorry about the phrasing - i'm looking at the problem completely from neutron logs - i (neutron) get port_update request with 2 different hosts in a span of less than 30secs. typically, there's about 3 port updates until its finally bound and in active state. i couldn't figure out where the aggresive timeout is, that forces a retry to another host. so i was checking in nova group if that was known :) | |
| 13:57:46 | pipesinpain | wolverineav: but this is a port binding request for the *same* instance, yes? | |
| 13:58:11 | wolverineav | pipesinpain: yes, correct. | |
| 13:58:49 | pipesinpain | wolverineav: that is *seriously* odd... | |
| 13:59:07 | pipesinpain | wolverineav: I can't understand why multiple hosts would be port-bound to the same instance at the same time | |
| 13:59:28 | pipesinpain | wolverineav: that would mean that the scheduler essentially picked multiple hosts for the same instance. | |
| 13:59:51 | pipesinpain | wolverineav: or that a retry is occurring nearly instantaneously. | |
| 14:00:31 | pipesinpain | wolverineav: since the scheduler absolutely cannot simultaneously pick multiple hosts for the same instance, it must be related to the retry. | |
| 14:00:49 | jmccarthy | pipesinpain: Hmm ok, what would that instance be booting from for future boots like after it's created ? I thought it did stuff with the image and used it to make the vol bootable like. I only see a /dev/sda and /dev/sda1 in the instance itself (which I thought was the new volume it created, made bootable) - I could be missing something .. | |
| 14:01:01 | pipesinpain | wolverineav: can you please look in the compute logs for "Retry"? | |
| 14:01:27 | pipesinpain | jmccarthy: it would be booting from the same original image you selected in horizon. | |
| 14:01:37 | pipesinpain | jmccarthy: and would *attach* the persistent data volume after boot. | |
| 14:01:38 | wolverineav | pipesinpain: i had to follow threads to figure out what was happening. it caused some inconsistency issue between neutron and DB and backend network controller when two requests got processed in different order in neutron vs calls made to controller. | |
| 14:02:06 | wolverineav | pipesinpain: yes, let me check. I did not notice anything earlier. but i'll enable debug to be sure | |
| 14:02:13 | jmccarthy | pipesinpain: Hmm ok that's still ok, I mean there is no other disk file for it in /var/lib/instances anywhere (at least that I could find) | |
| 14:02:23 | pipesinpain | wolverineav: any chance you can upload the compute logs somewhere? or at least, a section of them around where this is happening? | |
| 14:02:59 | jmccarthy | pipesinpain: (of whatever the path is, I'm sure I typed that wrong)# | |
| 14:03:40 | wolverineav | pipesinpain: it doesn't always happen, so i don't have debug enabled logs (first happened at a customer site). i'll ping back here (/mailing list) once i have that reproduced. | |
| 14:04:32 | pipesinpain | wolverineav: ++, thank you sir! :) | |
| 14:06:38 | jmccarthy | pipesinpain: Sorry yes of course - I am confusing the preparation of volume to be used to boot from - nevermind :) I still think there is a bug or something strange going on :) Thanks for your help ! | |
| 14:10:08 | mriedem | giblet: weren't we at one point talking about the scheduler creating the allocations against the nic bw providers? now that gets passed down to neutron through port binding in the latest revision of the spec. | |
| 14:10:17 | pipesinpain | jmccarthy: no problemo :) | |
| 14:10:19 | mriedem | honestly i can't remember which one i was pushing for a month ago | |
| 14:10:50 | giblet | mriedem: scheduler does the claim, but nova sends to neutron what was claimed | |
| 14:11:02 | mriedem | oh, ok, that was'nt clear to me yet | |
| 14:13:28 | giblet | mriedem: please leave a comment and I will clarify that in the spec | |
| 14:13:30 | Cardoe | mriedem: You asked about rebooting rescued instances the other day at Rackspace here. I'm trying to find you an answer. | |
| 14:13:54 | mriedem | Cardoe: that would be great, thanks - also posted to the openstack-dev ML | |
| 14:14:19 | Cardoe | full disclosure, I'm fairly new here and I don't have any OpenStack experience/knowledge. | |
| 14:14:50 | Cardoe | I'm purely a Xen hypervisor individual. But I hate hearing about downstream custom bits and fully advocate upstreaming all. | |
| 14:15:09 | Cardoe | Hence why someone prodded me, but I'll do my best to get to the bottom of it. | |
| 14:17:00 | jgwentworth | pipesinpain: just looked at those two bugs, agreed they look similar -- that is, the fix for the one should have avoided the problem in the other it seems. but I know not more than that, have to look deeper into it | |
| 14:24:15 | openstackgerrit | Eric Fried proposed openstack/nova master: Base test module/class for functional placement db https://review.openstack.org/564590 | |
| 14:24:16 | openstackgerrit | Eric Fried proposed openstack/nova master: Use test_base symbols directly https://review.openstack.org/564592 | |
| 14:24:17 | openstackgerrit | Eric Fried proposed openstack/nova master: Use helpers in test_resource_provider (func) https://review.openstack.org/564638 | |
| 14:25:27 | mriedem | pipesinpain: giblet: thinking about ironic and this nw bw qos thing - in the case of ironc, if the user wants guaranteed minimum bw, do they just request that using a specific ironic flavor with a custom resource class indicating that ironic node will have that min bw? | |
| 14:25:47 | mriedem | or its still tied up in the port? | |
| 14:27:02 | giblet | mriedem: I don't know. How does ironic request a port today? via neutron? | |
| 14:27:19 | mriedem | jroll: ^ | |
| 14:28:24 | giblet | mriedem, jroll: or maybe the real question is, who handles the compute side of the networking in case of ironic | |
| 14:29:04 | giblet | mriedem, jroll: as the current bandwidth spec only handles the bandwidth resource on the device that is in the compute node | |
| 14:29:35 | mriedem | giblet: the reason i ask is because i was reading the "Finding the compute RP" section | |
| 14:29:48 | mriedem | and thinking about how >1 ironic node will have the same compute 'host' | |
| 14:30:09 | kashyap | mriedem: When you get a sec, does this require a release note? Remove support for monitoring Intel CMT `perf` events https://review.openstack.org/565242 | |
| 14:30:11 | mriedem | if we just can't support ironic, fine, i just want to make sure i understand it's a limitation | |
| 14:30:19 | mriedem | kashyap: yes | |
| 14:30:23 | mriedem | at least as an fyi | |
| 14:30:35 | mriedem | kashyap: just put in an 'other' release note | |
| 14:30:40 | kashyap | mriedem: Yeah, thought so. Let me make it right away | |
| 14:30:41 | mriedem | or 'update' | |
| 14:30:42 | kashyap | Thanks | |
| 14:30:44 | mriedem | *'upgrade' | |
| 14:31:04 | kashyap | Yep, the 'upgrade' tag seems more applicable | |
| 14:31:26 | giblet | mriedem: the name field of the ResourceProvider has a unique constraint in the DB so the unique compute node RP name is enforced for ironic case as well | |
| 14:32:12 | giblet | mriedem: I guess ironic uses the node name instead of the host name for the compute RP | |
| 14:35:43 | Cardoe | mriedem: So vm_states.py doesn't allow rebooting from rescue either. | |
| 14:36:10 | jmccarthy | mriedem: Hiya, just wondering if you have any ideas maybe about this bug ? https://bugs.launchpad.net/nova/+bug/1769131 It seems like another bug that has come up before (1666831) but I'm not sure what the story is | |
| 14:36:10 | openstack | jmccarthy: Error: Could not gather data from Launchpad for bug #1769131 (https://launchpad.net/bugs/1769131). The error has been logged | |
| 14:36:10 | mriedem | Cardoe: correct. the upstream nova api doesn't allow rebooting a rescued vm | |
| 14:36:18 | Cardoe | mriedem: I'm talking about the rax patched version | |
| 14:36:31 | mriedem | Cardoe: oh, heh | |