| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-04 | |||
| 14:01:01 | pipesinpain | wolverineav: can you please look in the compute logs for "Retry"? | |
| 14:01:27 | pipesinpain | jmccarthy: it would be booting from the same original image you selected in horizon. | |
| 14:01:37 | pipesinpain | jmccarthy: and would *attach* the persistent data volume after boot. | |
| 14:01:38 | wolverineav | pipesinpain: i had to follow threads to figure out what was happening. it caused some inconsistency issue between neutron and DB and backend network controller when two requests got processed in different order in neutron vs calls made to controller. | |
| 14:02:06 | wolverineav | pipesinpain: yes, let me check. I did not notice anything earlier. but i'll enable debug to be sure | |
| 14:02:13 | jmccarthy | pipesinpain: Hmm ok that's still ok, I mean there is no other disk file for it in /var/lib/instances anywhere (at least that I could find) | |
| 14:02:23 | pipesinpain | wolverineav: any chance you can upload the compute logs somewhere? or at least, a section of them around where this is happening? | |
| 14:02:59 | jmccarthy | pipesinpain: (of whatever the path is, I'm sure I typed that wrong)# | |
| 14:03:40 | wolverineav | pipesinpain: it doesn't always happen, so i don't have debug enabled logs (first happened at a customer site). i'll ping back here (/mailing list) once i have that reproduced. | |
| 14:04:32 | pipesinpain | wolverineav: ++, thank you sir! :) | |
| 14:06:38 | jmccarthy | pipesinpain: Sorry yes of course - I am confusing the preparation of volume to be used to boot from - nevermind :) I still think there is a bug or something strange going on :) Thanks for your help ! | |
| 14:10:08 | mriedem | giblet: weren't we at one point talking about the scheduler creating the allocations against the nic bw providers? now that gets passed down to neutron through port binding in the latest revision of the spec. | |
| 14:10:17 | pipesinpain | jmccarthy: no problemo :) | |
| 14:10:19 | mriedem | honestly i can't remember which one i was pushing for a month ago | |
| 14:10:50 | giblet | mriedem: scheduler does the claim, but nova sends to neutron what was claimed | |
| 14:11:02 | mriedem | oh, ok, that was'nt clear to me yet | |
| 14:13:28 | giblet | mriedem: please leave a comment and I will clarify that in the spec | |
| 14:13:30 | Cardoe | mriedem: You asked about rebooting rescued instances the other day at Rackspace here. I'm trying to find you an answer. | |
| 14:13:54 | mriedem | Cardoe: that would be great, thanks - also posted to the openstack-dev ML | |
| 14:14:19 | Cardoe | full disclosure, I'm fairly new here and I don't have any OpenStack experience/knowledge. | |
| 14:14:50 | Cardoe | I'm purely a Xen hypervisor individual. But I hate hearing about downstream custom bits and fully advocate upstreaming all. | |
| 14:15:09 | Cardoe | Hence why someone prodded me, but I'll do my best to get to the bottom of it. | |
| 14:17:00 | jgwentworth | pipesinpain: just looked at those two bugs, agreed they look similar -- that is, the fix for the one should have avoided the problem in the other it seems. but I know not more than that, have to look deeper into it | |
| 14:24:15 | openstackgerrit | Eric Fried proposed openstack/nova master: Base test module/class for functional placement db https://review.openstack.org/564590 | |
| 14:24:16 | openstackgerrit | Eric Fried proposed openstack/nova master: Use test_base symbols directly https://review.openstack.org/564592 | |
| 14:24:17 | openstackgerrit | Eric Fried proposed openstack/nova master: Use helpers in test_resource_provider (func) https://review.openstack.org/564638 | |
| 14:25:27 | mriedem | pipesinpain: giblet: thinking about ironic and this nw bw qos thing - in the case of ironc, if the user wants guaranteed minimum bw, do they just request that using a specific ironic flavor with a custom resource class indicating that ironic node will have that min bw? | |
| 14:25:47 | mriedem | or its still tied up in the port? | |
| 14:27:02 | giblet | mriedem: I don't know. How does ironic request a port today? via neutron? | |
| 14:27:19 | mriedem | jroll: ^ | |
| 14:28:24 | giblet | mriedem, jroll: or maybe the real question is, who handles the compute side of the networking in case of ironic | |
| 14:29:04 | giblet | mriedem, jroll: as the current bandwidth spec only handles the bandwidth resource on the device that is in the compute node | |
| 14:29:35 | mriedem | giblet: the reason i ask is because i was reading the "Finding the compute RP" section | |
| 14:29:48 | mriedem | and thinking about how >1 ironic node will have the same compute 'host' | |
| 14:30:09 | kashyap | mriedem: When you get a sec, does this require a release note? Remove support for monitoring Intel CMT `perf` events https://review.openstack.org/565242 | |
| 14:30:11 | mriedem | if we just can't support ironic, fine, i just want to make sure i understand it's a limitation | |
| 14:30:19 | mriedem | kashyap: yes | |
| 14:30:23 | mriedem | at least as an fyi | |
| 14:30:35 | mriedem | kashyap: just put in an 'other' release note | |
| 14:30:40 | kashyap | mriedem: Yeah, thought so. Let me make it right away | |
| 14:30:41 | mriedem | or 'update' | |
| 14:30:42 | kashyap | Thanks | |
| 14:30:44 | mriedem | *'upgrade' | |
| 14:31:04 | kashyap | Yep, the 'upgrade' tag seems more applicable | |
| 14:31:26 | giblet | mriedem: the name field of the ResourceProvider has a unique constraint in the DB so the unique compute node RP name is enforced for ironic case as well | |
| 14:32:12 | giblet | mriedem: I guess ironic uses the node name instead of the host name for the compute RP | |
| 14:35:43 | Cardoe | mriedem: So vm_states.py doesn't allow rebooting from rescue either. | |
| 14:36:10 | jmccarthy | mriedem: Hiya, just wondering if you have any ideas maybe about this bug ? https://bugs.launchpad.net/nova/+bug/1769131 It seems like another bug that has come up before (1666831) but I'm not sure what the story is | |
| 14:36:10 | openstack | jmccarthy: Error: Could not gather data from Launchpad for bug #1769131 (https://launchpad.net/bugs/1769131). The error has been logged | |
| 14:36:10 | mriedem | Cardoe: correct. the upstream nova api doesn't allow rebooting a rescued vm | |
| 14:36:18 | Cardoe | mriedem: I'm talking about the rax patched version | |
| 14:36:31 | mriedem | Cardoe: oh, heh | |
| 14:36:56 | Cardoe | I had to figure out how to get into a compute node over here. | |
| 14:37:20 | mriedem | is it maybe hard-coded into the nova/compute/api.py code? | |
| 14:37:38 | mriedem | there was some refactoring done in there in the last couple of releases, and i don't think rax has updated code to match upstream in a long time | |
| 14:37:56 | Cardoe | checked that too. the @check_instance_state is the same as upstream | |
| 14:38:00 | mriedem | so don't check vm_states.py, check nova/compute/api.py:API.reboot() | |
| 14:38:02 | jroll | mriedem: giblet: for ironic, nova creates the port and ironic updates it later in the provisioning process | |
| 14:38:04 | mriedem | hmm | |
| 14:38:31 | jroll | I would like to see qos be a traits thing | |
| 14:38:45 | Cardoe | well not the same because yeah I see the results of a big refactor but it doesn't seem to be allowed in there. | |
| 14:38:45 | jroll | for ironic, it's just a property of the machine, not anything we can control | |
| 14:39:20 | mriedem | Cardoe: i don't suppose anyone still has a line to Matthew Sherborne huh | |
| 14:39:25 | giblet | jroll: the current spec defines bandwidth as a resource on provided by the physical device on the compute https://review.openstack.org/#/c/502306/ | |
| 14:39:35 | mriedem | because https://bugs.launchpad.net/nova/+bug/1170237 seems totally bogus | |
| 14:39:35 | openstack | Launchpad bug 1170237 in OpenStack Compute (nova) "cannot reboot instances when in rescue mode" [Medium,Fix released] - Assigned to Matthew Sherborne (msherborne+openstack) | |
| 14:39:35 | jroll | hrm | |
| 14:39:49 | jroll | giblet: I'll visit it when I'm back from errands | |
| 14:39:55 | mriedem | unless he just took a bunch of nova people on a wild goose chase 5 years ago | |
| 14:39:56 | giblet | jroll: thanks | |
| 14:40:22 | Cardoe | mriedem: I'm lighting up some folks. I'll get you a better answer soon. | |
| 14:45:00 | giblet | mriedem: as far as I understand neutron only configures TOR switches for ironic and ironic does the compute side physical device handling | |
| 14:45:20 | giblet | mriedem: this means that neutron does not know what capabilities the physical device has | |
| 14:45:55 | giblet | mriedem: so I think if ironic needs bandwidth handling the ironic virtdriver can create RPs with traits/resources and the user request that via flavor extra_spec | |
| 14:47:15 | openstackgerrit | Eric Fried proposed openstack/nova master: placement: Granular GET /allocation_candidates https://review.openstack.org/517757 | |
| 14:47:17 | giblet | mriedem: when (if) the bandwidth handling support is extened to TOR switches then baremetal neutron ports will have a resource request describing what the resource the port needs on the TOR switch | |
| 14:50:52 | kashyap | "Lighting up some folks" brings a very strong image to the brain... | |
| 14:57:15 | mriedem | jgwentworth: am i dreaming this up, or did we agree at the dublin ptg to add a type column to the consumers table in placement to be able to distinguish instance from migration consumers? | |
| 15:00:07 | jgwentworth | mriedem: no, there wasn't agreement on that. but I realized what I needed it for (quotas) could be achieved with instance_mappings + queued_for_delete column + user_id column | |
| 15:00:33 | mriedem | oh i was just looking for it in your "count quotas using placement" spec | |
| 15:01:16 | mriedem | ah i found it here https://etherpad.openstack.org/p/nova-ptg-rocky | |
| 15:01:24 | mriedem | "In Sydney we talked about tracking a 'type' in the placement allocations/consumers tab"... | |
| 15:01:35 | jgwentworth | yeah, that spec has been languishing and I really needed to update it after tssurya proposed her spec for adding queued_for_delete since mine would depend on that | |
| 15:02:31 | mriedem | giblet: i dont know what "If QoS aware and non QoS aware ports are mixed on the same physical port" means | |
| 15:02:58 | mriedem | that's possible today? to have a port in neutron that is both qos aware and not at the same time? | |
| 15:03:02 | jgwentworth | yeah, we discussed the 'type' idea a bit at the ptg but things got pretty complicated, it would not be straightforward. we may need it someday for other reasons but nothing right now really needs it. I had thought I needed it for quotas but realized I didn't if we could have two more instance_mappings columns | |
| 15:03:34 | mriedem | jgwentworth: ack; it's the new bdm.uuid column - something we'll always think we need every 4 months | |
| 15:03:55 | giblet | mriedem: assume there is a compute node with an SRIOV PF that provide VFs for neutron ports. | |
| 15:04:11 | jgwentworth | mriedem: heh yeah | |
| 15:04:12 | giblet | mriedem: after our spec that PF will also provide bandwidth as well | |
| 15:04:29 | jgwentworth | (I think we do have the bdm.uuid column now tho) | |
| 15:04:30 | giblet | mriedem: a neutron port might have a QoS policy rule attached | |
| 15:05:01 | mriedem | jgwentworth: yes we do | |
| 15:05:08 | giblet | mriedem: when we place a neutron port with QoS policy rule attached to the above PF we will consume some bandwidth as well | |
| 15:05:12 | mriedem | for mdbooth's local device serial series | |
| 15:05:15 | jgwentworth | need=TRUE | |
| 15:05:20 | jgwentworth | ah right | |
| 15:05:38 | jgwentworth | persistent cereal numbers | |
| 15:05:52 | giblet | mriedem: but if there is two neutron port one with QoS another without QoS ends up using VFs from the same PF then the minimum bandwidth rule cannot be garanteed | |
| 15:06:17 | mriedem | jgwentworth: ala https://images-na.ssl-images-amazon.com/images/I/51hDzZDOhdL.jpg ? | |